PaperlayerView exact revision on arXiv

Source-linked AI summary

Word Embedding Visualization Via Dictionary Learning

Juexiao Zhang, Yubei Chen, Brian Cheung, Bruno A Olshausen

arXiv:1910.03833v2cs.CLcs.LG

TL;DR

Word embeddings encode semantic and syntactic information, but their individual dimensions are often difficult to interpret. This paper uses dictionary learning to decompose them into meaningful word factors, then uses those factors to visualize vectors and improve analogy-task performance across embedding methods.

  • Problem

    Individual dimensions of most word embeddings are not semantically meaningful, making the learned representations difficult to visualize and interpret.

    Source: §1 · p. 1

  • Method

    Dictionary learning decomposes word vectors into sparse combinations of elementary factors, which are grouped by similar meaning and visualized through their activations.

    Source: §1 · p. 2, §2 · p. 2, §3.1 · p. 3

  • Results

    The authors report that learned factors have clear semantic meanings and consistently improve many word embedding methods on word analogy tasks.

    Source: §1 · p. 2, §5 · p. 8

  • Takeaways & Limitations

    Word factors provide a basis for visualizing word-vector compositions and developing analogy tasks beyond the existing 14.

    Source: §5 · p. 8

    The visualized word vectors come from methods that ignore the context of a word in a specific instance.

    Source: §5 · p. 8

Abstract

from arXiv · show

Co-occurrence statistics based word embedding techniques have proved to be very useful in extracting the semantic and syntactic representation of words as low dimensional continuous vectors. In this work, we discovered that dictionary learning can open up these word vectors as a linear combination of more elementary word factors. We demonstrate many of the learned factors have surprisingly strong semantic or syntactic meaning corresponding to the factors previously identified manually by human inspection. Thus dictionary learning provides a powerful visualization tool for understanding word embedding representations. Furthermore, we show that the word factors can help in identifying key semantic and syntactic differences in word analogy tasks and improve upon the state-of-the-art word embedding techniques in these tasks by a large margin.

1 Introduction

Word embeddings encode semantic and syntactic information, but their individual dimensions are often difficult to interpret, and existing visualization approaches have limitations. The paper uses dictionary learning to decompose vectors into elementary factors, visualize their compositions, and support analogy analysis and corpus-bias diagnosis.

1 Introduction

  • Individual embedding dimensions are often not semantically meaningful, while nearest neighbors, t-SNE, and subset PCA each have limitations for revealing word relationships.Nearest neighbors reduce relatedness to a scalar, t-SNE can distort linear structure, and subset PCA requires manually selected word sets.Source: §1 · pp. 1–2
  • Dictionary learning treats word vectors as sparse linear combinations of more elementary factors, extending earlier sparse-representation work toward visualization.The authors argue that binary coefficients or a fixed k-sparse setting are not ideal when words involve different numbers of factors with different strengths.Source: §1 · p. 2
  • The learned factors support visualizing word-vector compositions and developing new analogy tasks, while factor proportions can help diagnose corpus bias.The paper reports that factor proportions may change with the corpus used to train embeddings.Source: §1 · p. 2
  • Spectral clustering groups factors with similar meanings, and these groups help identify semantic differences in analogy tasks across embedding techniques.The authors report significant improvement in almost every analogy subcategory, irrespective of the embedding technique.Source: §1 · p. 2

2 Word Factors Learning and Spectral Grouping

The method learns word factors by representing word vectors with nonnegative overcomplete sparse coding, then groups semantically similar factors using spectral clustering. The clustering uses similarities derived from factor co-activations and a sparse, symmetrized adjacency matrix.

2 Word Factors Learning and Spectral Grouping

  • Nonnegative overcomplete sparse coding models each word vector as a sparse linear superposition of word factors, with the coefficient vector inferred from the representation.The method assumes a small independent-Gaussian noise vector and uses an overcomplete factor dictionary, typically with more factors than vector dimensions.Source: §2 · p. 2
  • Word vectors are sampled in proportion to corpus frequency, and iterative coding inference and factor updates learn factors that reconstruct them.The inverse problem is solved with FISTA; after sparse coefficients are inferred, factors are updated to improve reconstruction.Source: §2 · pp. 2–3
  • Spectral clustering groups factors because factors with similar meanings may co-activate when decomposing a word vector.The method computes a normalized covariance matrix of factor coefficients, then sparsifies it by retaining the q largest values in each row.Source: §2 · p. 3
  • The sparse factor-similarity matrix is symmetrized to form the adjacency matrix used to build a normalized Laplacian for clustering.The clustering pipeline normalizes rows of the Laplacian eigenvector matrix and applies k-means to those rows.Source: §2 · p. 3

3 Visualization

Dictionary learning decomposes word vectors into sparse combinations of interpretable factors, which can expose semantic and syntactic structure. The section illustrates factor meanings, vector decompositions, and differences across embedding models.

3 Visualization

  • Sparse coding represents each word vector as a linear combination of a limited number of learned factors, whose coefficients form its decomposition.The paper organizes all word vectors and their decompositions as matrices X and A.Source: §3 · p. 3

name.

  • Top-activation words guide semantic names for unsupervised factors, and their coefficients indicate how strongly each factor is active for a word.The authors name factor 59 “medical” because its high-activation words relate to medical purposes.Source: §3.1 · p. 3, Table 1 · p. 5

3.1 Word Factor Visualization

  • Learned factors match key differences in all 14 analogy tasks, while profession factors distinguish profession forms from their action-related words.The paper also uses remaining factors to suggest automatically generated analogy tasks.Source: §3.1 · p. 4
  • Fewer than 10% of factors remain unidentified; some activate densely or track word frequency, and co-activation can group similar factors for more reliable meaning detection.The authors suggest frequency-based optimization exposure as one possible explanation for some unclear factors.Source: §3.1 · pp. 4–5

3.2 Word Vector Visualization

  • Sparse factor combinations separate polysemous meanings and can mix semantic with syntactic information within one word vector.The “apple” vector includes technology, fruit, dessert, and organism factors; “big” combines size and comparative factors.Source: §3.2 · p. 5
  • Adding or subtracting factors from word vectors successfully modifies semantic and syntactic meanings, including part of speech, gender, and sentiment.Factors were scaled by 4 because their unit norm is shorter than the 7.2 average word-vector norm.Source: §3.2 · p. 6, Table 2 · p. 6

3.3 Comparison Across Different Models

  • Model comparisons reveal distinct patterns: fastText gives “sing” high present-tense activation through its “ing” subword, while “king” has a bedding factor in GloVe Crawl but not GloVe Wiki.The authors use factor visualizations to diagnose differences associated with embedding methods and training corpora.Source: §3.3 · pp. 6–7, Figure 6 · p. 7

4 Improvement in Semantic and Syntactic Tasks with Word Factors

The section contrasts vector-arithmetic errors with factor-group selection, reporting improvements across embedding methods and newly constructed analogy tasks.

4 Improvement in Semantic and Syntactic Tasks with Word Factors

  • Vector arithmetic can return a semantically close answer while missing the key relationship asked by an analogy.The conventional method predicts a word by finding the nearest neighbor to B − A + C.Source: §4 · p. 7
  • Factor-group selection improved every embedding algorithm on almost every subtask, though three Table 3 experiments declined, likely because clustering separated relevant factors.The authors attribute the decreases most likely to correct factors not being grouped together during spectral clustering.Source: §4 · p. 7, Table 3 · p. 8
  • The selection method requires a predicted word to have higher activation on the relevant factor groups than words in the other category.For example, a candidate “queen” must activate the “female” factor group more than the male-category words.Source: §4 · p. 7
  • Consistent improvements also appeared on new analogy tasks, including ideology, profession, and adj-to-verb relationships.The reported behavior for these tasks matches that of Table 3.Source: §4 · p. 7, Table 4 · p. 8

5 Discussion

The discussion concludes that dictionary learning extracts interpretable factors and supports analogy-task improvements, while identifying contextualized embeddings as future work.

5 Discussion

  • The authors conclude that dictionary learning extracts elementary factors, enables meaningful vector decompositions and visualizations, and consistently improves many embedding methods on analogy tasks.They suggest factors may support analogy tasks beyond the existing 14.Source: §5 · p. 8
  • The reason factors combine linearly remains unresolved; the authors hypothesize they reflect subcategories in words’ context statistics.They cite sparse word representations as one possible explanation for the linear combinations.Source: §5 · p. 8
  • The visualized vectors come from methods that ignore instance-specific context; applying the approach to contextualized and attention-based vectors is future work.The authors report preliminary generalization to different transformer models.Source: §5 · p. 8

A The Word Factor Naming Procedure

Factors are named from frequently occurring words with strong activations; the examples span semantic and syntactic meanings.

A The Word Factor Naming Procedure

  • For each factor, word frequency weights activation, and the top words contributing 20% of total weighted activation guide its name.The number of selected words varies by factor and is usually up to 200.Source: A The Word Factor Naming Procedure · p. 11
  • The authors present “national” and “mobile&IT” as semantic factors, naming them from activated words associated with national identities and mobile devices or IT.The “national” example includes nationality terms; “mobile&IT” includes device, brand, and technology terms.Source: A The Word Factor Naming Procedure · p. 11
  • The “ideology” factor is illustrated by highly activating terms such as nationalism, liberalism, socialism, and communism.The authors identify the shared ideological meaning from the factor’s activated words.Source: A The Word Factor Naming Procedure · pp. 11–12

B The Details of the Non-negative Sparse Coding Optimization

The optimization learns a 1,000-factor dictionary for 300-dimensional vectors through iterative sparse coding and dictionary updates.

B The Details of the Non-negative Sparse Coding Optimization

  • The experiments use 300-dimensional word vectors and a dictionary of 1,000 word factors.The authors note that other settings produced qualitatively similar results.Source: B The Details of the Non-negative… · p. 12
  • Sparse-coefficient optimization uses minibatches of 100 vectors, converges in 500 FISTA steps, and uses λ = 0.5 from experiments over 0.3 to 1.After inferring sparse coefficients, the dictionary is updated based on Optimization 10.Source: B The Details of the Non-negative… · p. 12
  • The dictionary update uses an approximate second-order method with a diagonal Hessian approximation; training runs 200,000 steps and takes about 2–3 hours on an Nvidia 1080 Ti GPU.The authors say this update usually leads to much faster convergence of the word factors.Source: B The Details of the Non-negative… · p. 12

C The New Word Analogy Tasks Generated

The section uses generated adj-to-verb and ideology examples to demonstrate that word factors are more elementary structures than word vectors.

C The New Word Analogy Tasks Generated

D Factor Group Co-activation

Heat maps show that factors within a group often co-activate on the same words, appearing as parallel bright bands.

D Factor Group Co-activation

E Word Factors and Relatedness

Table E.1 compares word factors with nearby words in the embedding space to show that the discovered factors are not simply existing words.

E Word Factors and Relatedness

  • Table E.1 compares factors’ top-activation words with their nearest embedding-space words and neighbors, showing factors are not exactly existing words.Source: Table E.1 · p. 14
Loading 1910.03833v2…