PaperlayerView exact revision on arXiv

Source-linked AI summary

Transformer visualization via dictionary learning: contextualized embedding as a linear superposition of transformer factors

Zeyu Yun, Yubei Chen, Bruno A Olshausen, Yann LeCun

arXiv:2103.15949v2cs.CLcs.LG

TL;DR

Transformer representation learning remains difficult to interpret, with a shortage of tools for detailed visualization. This paper uses dictionary learning to represent contextualized embeddings as superpositions of transformer factors, and reports visualized hierarchical patterns including word-level disambiguation, sentence-level pattern formation, and long-range dependency.

  • Problem

    The authors identify limited understanding of transformer representations and a lack of tools for detailed visualization; probing tasks can be too simple to reflect a model’s true capacity.

    Source: §1 · p. 1

  • Method

    The paper applies dictionary learning to model contextualized word embeddings as sparse superpositions of transformer factors, using a shared dictionary across layers.

    Source: §2 · p. 2

  • Results

    The visualizations show word-level disambiguation, sentence-level pattern formation, and long-range dependency in transformer representations.

    Source: §4 · p. 8

  • Takeaways & Limitations

    The authors conclude that learned transformer factors can provide a visualization tool for examining hierarchical semantic or syntactic representations at different stages.

    Source: §4 · p. 8

    The approach assumes contextualized embeddings can be represented as sparse combinations of transformer factors and uses one dictionary across layers based on the skip-connection hypothesis.

    Source: §2 · p. 2

Abstract

from arXiv · show

Transformer networks have revolutionized NLP representation learning since they were introduced. Though a great effort has been made to explain the representation in transformers, it is widely recognized that our understanding is not sufficient. One important reason is that there lack enough visualization tools for detailed analysis. In this paper, we propose to use dictionary learning to open up these "black boxes" as linear superpositions of transformer factors. Through visualization, we demonstrate the hierarchical semantic structures captured by the transformer factors, e.g., word-level polysemy disambiguation, sentence-level pattern formation, and long-range dependency. While some of these patterns confirm the conventional prior linguistic knowledge, the rest are relatively unexpected, which may provide new insights. We hope this visualization tool can bring further knowledge and a better understanding of how transformer networks work. The code is available at https://github.com/zeyuyun1/TransformerVis

1 Introduction

Transformer interpretation remains limited, and probing tasks offer only partial insight into what models learn and where representations arise. The paper proposes dictionary learning as a visualization approach to complement those techniques.

1 Introduction

  • Transformer interpretation remains limited despite growing visualization and analysis efforts.Existing approaches include attention visualization and probing classifiers trained for tasks such as POS tagging and NER tagging.Source: §1 · p. 1
  • Probing tasks such as POS and NER tagging may be too simple to reflect a model’s true capacity.The cited discussion also notes that probing verifies prior structures but cannot reveal structures beyond prior knowledge or locate where representations are learned.Source: §1 · p. 1
  • Contextualized embeddings of the same word separate into different clusters in UMAP visualizations, while similarity analyses also find context-specific representations.These findings are reported by Reif et al. (2019) and Ethayarajh (2019), respectively.Source: §1 · p. 1
  • Dictionary learning has been used to explain, improve, and visualize uncontextualized word embeddings.The cited works are Faruqui et al. (2015), Arora et al. (2018), and Zhang et al. (2019).Source: §1 · p. 1
  • Dictionary learning is proposed to visualize transformer representations and supplement probing-task methods.The authors report that learned factors can reveal linguistic structures and surprising insights.Source: §1 · p. 2

2 Method

The method treats contextualized word embeddings as sparse superpositions of transformer factors and learns one dictionary across layers. It visualizes factors through their strongest activations and uses layer-wise importance curves to estimate when factors emerge.

2 Method

  • The method hypothesizes that contextualized word embeddings are sparse linear superpositions of elementary transformer factors.The factors are proposed as elementary elements in the representation.Source: §2 · p. 2
  • A single dictionary is learned across layers because skip connections are hypothesized to make each layer a superposition of lower-layer representations.The block output is described as the sum of modifications along the way.Source: §2 · p. 2
  • The method collects contextualized word embeddings from each layer into one set and models each vector with sparse nonnegative factor coefficients.The dictionary may be overcomplete, with small independent Gaussian noise; FISTA is used to solve the inverse problem.Source: §2 · p. 2
  • Top-activation sequences are used to visualize factors, with LIME assigning token weights to estimate their relative importance.The method ranks the 1,000 contextualized vectors with largest sparse coefficients for each factor.Source: §2 · pp. 2–3
  • A factor’s importance score averages its 1,000 largest sparse coefficients at each layer to estimate where it emerges.Earlier peaks may indicate lower-level meanings, while later peaks may indicate mid- or high-level structures.Source: §2 · p. 3
  • Relatively smooth importance curves suggest early features persist with decaying weight while higher-layer abstract information develops gradually.The passage points to Figures 3 and 5 as confirming this pattern.Source: §2 · p. 3

3 Experiments and Discoveries

Using a frozen 12-layer BERT and a single dictionary across layers, the experiments visualize transformer factors associated with word-level disambiguation, sentence-level patterns, and long-range dependencies.

Experiments and discoveries

  • Only a small number of factors are shown in the paper; an interactive website is offered to let readers explore more results and address cherry-picking concerns.The authors also provide more examples in appendices.Source: §3 · pp. 3, 8

Low-level: word-level polysemy disambiguation

  • By layer 4, the single feature Φ:,30 nearly linearly separates verb “left” from other senses and reportedly outperforms two supervised classifiers.Its activations move from blending the senses in layer 0 to increased separation in layers 2 and 4.Source: §3 · pp. 4–5, Figure 4 · p. 5

Mid level: sentence-level pattern formation

  • Mid-level factors capture phrase and sentence patterns: consecutive adjectives fully develop by layer 8, while unit-exchange substructures emerge in layers 4 and 6.The complete unit-exchange pattern appears at layer 8.Source: §3 · pp. 5–6, Figure 5 · p. 5, Figure 6 · p. 6
  • The authors report that most mid-level patterns begin developing at layer 4 or 6, with large shares of top activations matching the described patterns.Their analysis records pattern matches among the top 200 activated words and contexts across layers.Source: §3 · pp. 6–7, Table 3 · p. 6

High-level: long-range dependency

  • To identify token contributions to factor activations, LIME fits a linear model on randomly masked sequence variants and treats its weights as a saliency map.The method uses the token weights to indicate positive or negative contributions and their magnitude.Source: §3 · p. 7
  • For Φ:,297, highly activated contexts contain repeated text sequences, a more abstract pattern than the mid-level factors.The examples repeat one snippet twice and another three times.Source: §3 · p. 8, Figure 7 · p. 8
  • Another high-level factor activates for two or more nearly identical embedding vectors at layer 10, irrespective of their content; its relation to capsule networks remains a possibility.The authors identify studying self-attention's role in this behavior as future work.Source: §3 · p. 8
  • A biography-beginning factor combines salient name, age, career, and family-relation information into a higher-level pattern.The visualized high-activation contexts all contain the beginning of a biography.Source: §3 · p. 8, Figure 8 · p. 8

4 Discussion

The discussion presents dictionary learning as a visualization tool for transformer representations and reports hierarchical semantic or syntactic patterns across layers. It suggests the approach may also help visualize other neural networks with skip connections.

Discussion

  • Dictionary learning is presented as a way to visualize transformer representations and supplement probing-task methods.The authors describe the learned factors as relatively reliable while also revealing surprising linguistic structures.Source: §4 · p. 8
  • The reported patterns span word-level disambiguation, sentence-level pattern formation, and long-range dependency across different stages.The authors relate early-layer low-level and later-layer abstract features to visualizations in CNNs.Source: §4 · p. 8
  • The authors suggest dictionary learning may help visualize other skip-connection networks, including ResNet and ViT models.Source: §4 · p. 8

A Importance Score (IS) Curves

Importance-score curves distinguish low-level factors from mid- and high-level factors by where their peaks occur; the appendix details LIME explanations, sparse-coding optimization, and links to additional visualizations.

A Importance Score (IS) Curves

  • Low-level factors peak in early layers and decline, whereas mid- and high-level factors rise toward peaks in later layers; individual vector dimensions show no such pattern.Figure 9 shows examples of low- and mid-level factor curves; Figure 10 contrasts them with dimension-level curves without sparse coding.Source: A Importance Score (IS) Curves · p. 11, Figure 9 · p. 11

B LIME: Local Interpretable Model-Agnostic Explanations

  • LIME masks random token positions with [UNK], fits a weighted linear model to estimate token contributions, and treats larger regression weights as greater saliency.The method can run feature selection by selecting the largest absolute weights and rerunning LIME on those positions; negative weights mark contexts that behave opposite to the factor pattern.Source: B LIME: Local Interpretable… · pp. 11–12

C The Details of the Non-negative Sparse Coding Optimization

  • The optimization concatenates 768-dimensional BERT-base embeddings, uses 200-vector minibatches and inverse-frequency weighting, and trains with λ = 0.27 for 200k steps.The authors report using 1000 FISTA steps for coefficient inference and diagonal-Hessian approximate second-order dictionary updates; training took roughly two days on an Nvidia 1080 Ti GPU.Source: C The Details of the Non-negative… · p. 12

D Hyperlinks for More Transformer Visualization

E Low-Level Transformer Factors

The low-level examples illustrate a factor labeled “mind” through varied sentence contexts, followed by examples for “Park” as a name.

E Low-Level Transformer Factors

  • The mind factor is labeled with a cognitive meaning and appears in varied contexts, including “make up my mind,” “on my mind,” and references to a mind in one’s head.The examples span idiomatic phrases and descriptions of thoughts or experiences.Source: E Low-Level Transformer Factors · p. 13

Explaination: Park: noun, ’park’ as the name

Explaination: left: verb, leaving, exiting

Examples for the “left” factor show uses of the verb associated with departing or leaving something behind.

Explaination: left: verb, leaving, exiting

  • Examples include people who departed, a story left unfinished, and a will that left an inheritance to someone.The examples illustrate leaving a place or leaving something in a particular state or to another person.Source: Explaination: left: verb, leaving,… · pp. 13–14

Transformer factor 33 in layer 4

The layer-4 factor labeled “light” is illustrated with contexts involving illumination and figurative uses of the word.

Transformer factor 33 in layer 4

  • Examples range from physical illumination, such as filming with little light or fading light, to figurative descriptions such as portraying someone in a sympathetic light.The examples also include compounds and names containing “light,” such as light-filled and early light.Source: Transformer factor 33 in layer 4 · p. 14

Explaination: plants: noun, vegetation

The visualized factors are illustrated through examples spanning vegetation-related terms, unit exchanges, male names, and attributive clauses.

Explaination: plants: noun, vegetation

Transformer factor 13 in layer 10 Explaination: Unit exchange with parentheses: e.g. 10 m (1000cm)

  • Numerical examples pair measurements with conversions or equivalent units, including distances, weights, and speeds.The examples include paired metric and imperial measurements.Source: Transformer factor 13 in layer 10… · p. 16

Transformer factor 24 in layer 10 Explaination: Male name

  • Examples for the male-name factor include references to Robert H. Tucker, Jr., and other people named with “jr.”The examples include both name mentions and descriptions of relationships or actions involving those people.Source: Transformer factor 24 in layer 10… · p. 16

Transformer factor 25 in layer 10 Explaination: Attributive Clauses

  • Attributive-clause examples include clauses introduced by “which” and clauses describing rankings, quantities, or stages.The examples include clauses modifying nouns such as “river,” “ton,” and “stage.”Source: Transformer factor 25 in layer 10… · pp. 16–17

Explaination: Some kind of disaster, something unfortunate happened

The examples associated with this factor describe adverse events, including injury, illness, fire, and death.

Explaination: Some kind of disaster, something unfortunate happened

  • Examples include a broken foot, natural deaths, typhus symptoms, a fire, and illness or death in office.Other snippets describe a home destroyed by fire and a person falling ill before dying in office.Source: Explaination: Some kind of disaster,… · p. 17

Transformer factor 50 in layer 10 Explaination: Doing something again, or making something new again

Examples associated with this factor include things being revamped, redesigned, or substantially overhauled.

Transformer factor 50 in layer 10 Explaination: Doing something again, or making something new again

  • Examples include a show revamp, a redesigned medal ribbon, and substantially overhauled towers.Other examples describe story mode being revamped and an area redevelopment receiving increased emphasis.Source: Transformer factor 50 in layer 10… · pp. 17–18

Transformer factor 51 in layer 10 Explaination: apostrophe s, possesive

The examples illustrate factors associated with possessive apostrophe-s, season-year spans, and past-tense forms.

Transformer factor 51 in layer 10 Explaination: apostrophe s, possesive

  • Possessive apostrophe-s examples include references to a book’s text, a song’s production, and a game’s writing.The examples also include possessive forms referring to a novel, album, and song.Source: Transformer factor 51 in layer 10… · p. 18

Explaination: Pattern: Consecutive years, this is convention to name foodball/rugby game season

Transformer factor 99 in layer 10 Explaination: past tense

  • Past-tense examples include “ranked,” “released,” “rated,” and “considered.”The snippets use these forms to report actions or judgments that occurred in the past.Source: Transformer factor 99 in layer 10… · p. 19

Transformer factor 102 in layer 10 Explaination: African name

The visualized factor examples cover African names, descriptive paraphrases, transition sentences, and references to locations.

Transformer factor 102 in layer 10 Explaination: African name

Transformer factor 125 in layer 10 Explaination: Describing someone in a paraphrasing style. Name, Career

Transformer factor 134 in layer 10 Explaination: Transition sentence

Transformer factor 152 in layer 10 Explaination: in some locations

Transformer factor 297 in layer 10 with saliency map Explaination: repetitive structure detector

The section presents visualization excerpts around factor 297 and varied article contexts, including recurring textual sequences and fragmented war- and music-production material; one excerpt provides only a navigation cue rather than explaining the factor.

Transformer factor 297 in layer 10 with saliency map Explaination: repetitive structure detector

  • Anthology drama CD titles recur in a sequence of similarly named Tales of Graces releases.The excerpts list multiple seasonal releases under the Anthology Drama CD naming pattern.Source: Transformer factor 297 in layer 10… · p. 22
  • Passenger and cardholder categories recur across adjacent fragments listing Star Alliance and EVA Air eligibility groups.The listed groups repeat passenger classes and cardholder categories across the fragments.Source: Transformer factor 297 in layer 10… · p. 22
  • A bibliographic sequence repeats the “book in vocal music” title pattern across numbered volumes.The excerpts show books 1 through 4 in the Modern Music Series.Source: Transformer factor 297 in layer 10… · p. 22
  • Factor 297’s examples span fragments about fish regulations, biographies, science, baseball, airline routes, and a gubernatorial election.The excerpts include material about trout-season restrictions, Marjory Stoneman Douglas, Babe Ruth, and William Goebel’s election dispute.Source: Transformer factor 297 in layer 10… · pp. 25–26
  • A highlighted fragment continues an account of Powell criticizing Lincoln’s administration and senators seeking his expulsion.Source: Transformer factor 297 in layer 10… · p. 26

Transformer factor 386 in layer 10 with saliency map Explaination: topic: war

Transformer factor 170 in layer 10 with saliency map Explaination: topic: music production

  • Music-production contexts range from music-video descriptions and artist or producer comments to vocal and performance material.The excerpts mention a music video, discussion of a song’s production, and performers’ training and appearances.Source: Transformer factor 170 in layer 10… · pp. 31–32
  • One music-production fragment says mixing was largely settled during production and completed within a few hours.The speaker describes many tracks and says the production helped nail much of the mixing.Source: Transformer factor 170 in layer 10… · p. 32

Transformer factor 297 in layer 10 with saliency map Explaination: repetitive structure detector

Loading 2103.15949v2…