PaperlayerView exact revision on arXiv

Source-linked AI summary

The geometry of hidden representations of large transformer models

Lucrezia Valeriani, Diego Doimo, Francesca Cuturello, Alessandro Laio, Alessio Ansuini, Alberto Cazzaniga

arXiv:2302.00294v2cs.LGcs.AIstat.ML

TL;DR

Because the final representation in self-supervised models generally supports reconstruction, the most semantically rich representation may be intermediate. Tracking intrinsic dimension and neighborhood structure across protein and image transformers, the authors find low-ID regions associated with abstract content and propose ID minima as an unsupervised layer-selection cue.

  • Problem

    In self-supervised models, the final representation generally supports input reconstruction, so the most semantically rich representation may lie in an intermediate layer.

    Source: §1 · p. 2

  • Method

    The authors track intrinsic dimension and neighborhood structure across transformer layers, using overlap with ground-truth labels as a semantic probe.

    Source: §2.2 · p. 3, §4 · p. 8

  • Results

    Across the studied architectures, intrinsic-dimension expansion precedes compression into low-ID representations rich in abstract content, while output-side neighbor rearrangement accompanies degradation of that content.

    Source: §4 · p. 8

  • Takeaways & Limitations

    ID-profile minima provide an unsupervised way to identify layers useful for downstream tasks such as protein homology searches and image classification.

    Source: §1 · p. 2

    The analysis could be reinforced by comparing minimum-ID representations with representations from supervised models, to clarify the iGPT second peak and the pLM plateau.

    Source: §4 · p. 8

Abstract

from arXiv · show

Large transformers are powerful architectures used for self-supervised data analysis across various data types, including protein sequences, images, and text. In these models, the semantic structure of the dataset emerges from a sequence of transformations between one representation and the next. We characterize the geometric and statistical properties of these representations and how they change as we move through the layers. By analyzing the intrinsic dimension (ID) and neighbor composition, we find that the representations evolve similarly in transformers trained on protein language tasks and image reconstruction tasks. In the first layers, the data manifold expands, becoming high-dimensional, and then contracts significantly in the intermediate layers. In the last part of the model, the ID remains approximately constant or forms a second shallow peak. We show that the semantic information of the dataset is better expressed at the end of the first peak, and this phenomenon can be observed across many models trained on diverse datasets. Based on our findings, we point out an explicit strategy to identify, without supervision, the layers that maximize semantic content: representations at intermediate layers corresponding to a relative minimum of the ID profile are more suitable for downstream learning tasks.

1 Introduction

Because a self-supervised model’s final representation generally supports reconstruction, its richest semantic representation may occur in an intermediate layer. The paper studies how transformer representations change across layers and proposes using ID profiles to select semantically meaningful layers without supervision.

1 Introduction

  • Across the studied transformers, representation changes form distinct phases revealed by simultaneous shifts in global ID and local neighborhood composition.Source: §1 · p. 2
  • After a prominent ID peak and neighbor rearrangement, relatively low-ID regions encode abstract information such as image class labels and protein remote homology.This information emerges gradually as ID decreases.Source: §1 · p. 2
  • The authors propose using the ID profile to identify, without supervision, layers whose representations are useful for downstream tasks such as protein homology searches or image classification.Source: §1 · p. 2
  • The observed sequence suggests a three-phase reconstruction strategy: expansion into high intrinsic dimension, compression into semantically meaningful representations, and decoding from the compressed representation.The authors note that this approach may help identify semantically rich representations when annotations are scarce.Source: §1 · p. 2

2 Methods

The methods quantify representation geometry and neighbor structure across transformer layers, then apply these analyses to protein and image models and datasets.

2.1 Intrinsic dimension.

  • Intrinsic dimension is estimated with TwoNN from each point’s distances to its first two nearest neighbors, under a locally constant density assumption.The ratio of those distances is modeled with a Pareto distribution whose shape parameter equals ID.Source: §2.1 · p. 2

2.2 Neighborhood overlap.

  • Neighborhood overlap measures shared k-nearest neighbors between representations, while overlap with ground-truth classes serves as a semantic probe.The analysis uses k = 10 for SCOPe proteins and k = 30 for ImageNet; the qualitative trend is reported as robust to changes in k.Source: §2.2 · p. 3

2.3 Models and Datasets

  • The analysis applies layerwise ID and overlap measurements to ESM2 protein models and iGPT image models, using pooled sequence representations and protein and ImageNet datasets.Protein analyses use ProteinNet for ID curves and consecutive-layer overlap, and a filtered SCOPe set of 10,256 sequences across 288 superfamilies for remote homology; image analysis uses 90,000 images from 300 ImageNet classes.Source: §2.3 · p. 3

3 Results

Across protein language models and image transformers, intrinsic-dimension (ID) profiles show an early expansion followed by lower-dimensional intermediate representations. These intermediate regions carry semantic information, and the ID profile identifies where it is most pronounced.

3.1 The intrinsic dimension profile in large transformers is similar in different tasks and architectures

  • Protein language models peak early (about 20, 25, and 32 for ESM-2 35M, 650M, and 3B), contract to a plateau between 5 and 7, then ascend; iGPT architectures share a local minimum of 22 and form a shallower second peak near the network’s end.Source: §3.1 · pp. 4–5
  • Neighborhood rearrangements are substantial during the early ID peak and final layers, but slower in the plateau, where larger protein models share over 90% of consecutive-layer neighbors.Image-transformer transitions from rapid to slower rearrangement are more gradual than in protein language models.Source: §3.1 · p. 5, Figure 2 · p. 6
  • During training, ESM-2 first develops an early ID peak and later compresses plateau representations, while iGPT-L also develops a local minimum and a second peak.The authors relate ESM-2's plateau compression to the emergence of semantic information.Source: §3.1 · p. 6, Figure 3 · p. 7

3.2 The intermediate representations of large transformers are the most semantically rich

  • The ID profile can identify intermediate layers with the most pronounced semantic information: the protein plateau and the iGPT relative minimum after the first peak.The protein plateau encodes remote homology, while the iGPT minimum corresponds to ImageNet-label information.Source: §3.1 · p. 5, §3.2 · pp. 6–7
  • Remote-homology information is highest in the protein-model plateau, while ImageNet-label overlap peaks near relative depth 0.4, where ID is lower.For proteins, nearest-neighbor predictive power is 0.8 in the plateau versus 0.4 in the last hidden layer.Source: §3.2 · p. 7

4 Discussion and conclusions

The discussion relates representation geometry to semantic content and proposes unsupervised identification of semantically rich layers. It describes transformer behavior as encoding into abstract representations and then decoding, while noting open questions and implementation choices.

4 Discussion and conclusions

  • The largest iGPT resembles a symmetric autoencoder, with a second ID peak and an almost symmetric pattern of overlap with ground-truth labels.The second peak approximately mirrors the first.Source: §4 · p. 8
  • Across studied models, intrinsic-dimension expansion and neighbor rearrangement precede compression; low-ID representations are quantitatively associated with abstract content, while output-near rearrangement accompanies its degradation.The authors relate output-near rearrangement to decoding.Source: §4 · p. 8
  • Comparing ID-minimum representations with those from supervised models could reinforce the analysis and clarify low-ID plateaus and the second iGPT peak.Source: §4 · p. 8
  • Average pooling is used for distance comparisons; on CIFAR10, it lowers ID without qualitatively changing its profile, while extracting after attention maps or MLP blocks leaves profiles quantitatively unchanged.Pooling also makes distance calculations computationally feasible for large image datasets.Source: §4 · pp. 8–9
  • ID profiles differ across ESM and iGPT, but local ID minima correspond to layers where a semantic property is most pronounced.The authors relate profile differences to modality, preprocessing, and training objectives.Source: §4 · p. 9
  • In preliminary NLP analyses, Llama-v2 70B has three ID peaks and two local minima, with the greatest sentiment-class overlap at the first minimum.The authors state that language requires diverse tasks and probes for comprehensive understanding.Source: §4 · p. 9

A Architecture details

The appendix describes how protein and image inputs are represented and processed through transformer blocks, with a final head supporting reconstruction.

A Architecture details

  • The protein-model table reports characteristics of the models used to extract representations, while protein inputs are encoded as token embeddings and processed through self-attention blocks.The final language-model head projects to a sequence of amino-acid dimensions.Source: A Architecture details · p. 14, Table 1 · p. 14
  • For iGPT, images are quantized into color codes and processed as ordered pixel sequences through attention blocks before the output head predicts pixel distributions.Source: A Architecture details · p. 14
  • Images are generated by sampling from p(x_i) one pixel at a time.Source: A Architecture details · p. 14

B Details about the intrinsic dimension estimation.

The appendix outlines the TwoNN intrinsic-dimension estimator, its scale-selection procedure, and a check of the local-density assumption underlying its use.

B Details about the intrinsic dimension estimation.

  • TwoNN estimates ID from each point’s distances to its first and second nearest neighbors, whose ratio follows a Pareto distribution under a Poisson point-process assumption.Source: B Details about the intrinsic… · p. 15
  • ID is estimated as the slope of a regression using sorted neighbor-distance ratios and their empirical cumulative probabilities.Source: B Details about the intrinsic… · p. 15
  • To limit noise-driven ID overestimation, the method selects a value from a plateau in estimates across progressively smaller random subsets; the study reports TwoNN estimates after fourfold decimation.The study found TwoNN less scale-dependent across dataset sizes from N/2 to N/8.Source: B Details about the intrinsic… · p. 15
  • PAk indicates that local density is constant on average within the first six neighbors, supporting the local-density assumption at the first-two-neighbor scale used for ID estimation.Source: B Details about the intrinsic… · p. 15

C Intrinsic dimension and overlap profiles on other protein language models.

Supplementary analyses show that the three-phase ID profile persists across protein language models with differing architectures and training datasets. Plateau-layer representations also improve a protein homology-search result without further training.

C Intrinsic dimension and overlap profiles on other protein language models.

  • Across ESM1b, ProtT5-XL-U50, ESM1v, and ProtBert, ProteinNet ID profiles retain an initial peak, a low-ID plateau, and a later rise despite differences in architecture and training datasets.Source: Figure S1 · p. 16
  • 1-kNN homology-search accuracy improves by approximately 6% on a plateau layer versus the last pre-output layer, without further training.The evaluation uses SCOPe protein domains from superfamilies with at least two sequences.Source: Figure S2 · p. 16

D Impact of the implementation details and hyperparameter choices.

The supplementary analyses test whether intrinsic-dimension and neighborhood-overlap profiles depend on representation choice, pooling, or neighborhood size. Profile shapes are qualitatively robust, although pooling shifts ID downward and large neighborhoods lower some ESM-2 overlap values.

D Impact of the implementation details and hyperparameter choices.

  • For CIFAR10 in Wide-ResNet28-8, average pooling preserves the qualitative ID-profile shape but shifts ID downward, especially in early stages.The shift is attributed likely to the first block's 16 channels versus 128, 256, and 512 in later blocks; qualitative robustness is expected when channel count substantially exceeds ID.Source: Figure S3 · p. 17
  • For ImageNet in iGPT Large, ID and label-overlap profiles are consistent after the first normalization layer and after self-attention maps.This supports robustness to the choice of layer representation.Source: Figure S4 · p. 17
  • Changing the neighborhood size k from 1 to 50 leaves the overlap profiles qualitatively unchanged in ESM-2 and iGPT Large.For ESM-2, lower overlap values at k = 50 reflect some superfamilies having fewer than 50 elements.Source: Figure S5 · p. 18

E Preliminary analysis of the intrinsic dimension and overlap in LLama2

For Llama2-70b on SST, the first local minimum of the intrinsic-dimension profile occurs at 0.3 relative depth, where sentiment-label overlap is highest.

E Preliminary analysis of the intrinsic dimension and overlap in LLama2

  • At 0.3 relative depth, sentiment-label overlap is highest, coinciding with the first local minimum in the ID profile.The figure plots ID against normalized block depth and ground-truth sentiment overlap at each self-attention block.Source: Figure S6 · p. 19
  • The SST representations' ID profile has a first local minimum at 0.3 relative depth in Llama2-70b.Source: Figure S6 · p. 19
  • The bottom panel tracks overlap with ground-truth sentence sentiment across self-attention blocks, and its maximum aligns with that ID minimum.Source: Figure S6 · p. 19

Related papers

Loading 2302.00294v2…