Source-linked AI summary
SemGloVe: Semantic Co-occurrences for GloVe from BERT
Leilei Gan, Zhiyang Teng, Yue Zhang, Linchao Zhu, Fei Wu, Yi Yang
TL;DR
GloVe’s local-window co-occurrence counts can miss relevant word pairs and use position-based weights that do not directly reflect semantic similarity. SemGloVe distills semantic co-occurrences from BERT using self-attention or masked-language-model outputs, and it outperforms GloVe across intrinsic and extrinsic evaluations. The method therefore provides a stronger static embedding alternative while retaining the efficiency advantages of noncontextualized representations.
Problem
GloVe’s predefined local context window limits extracted word pairs, while position-based weighting may not directly measure semantic relevance.
Method
SemGloVe replaces GloVe’s hard co-occurrence counts with BERT-derived semantic co-occurrences from self-attention weights or masked-language-model logits.
Results
SemGloVe outperforms GloVe on word similarity datasets and external Chunking, POS tagging, and NER tasks, with better averaged results than existing state-of-the-art noncontextualized methods.
Takeaways & Limitations
SemGloVe enriches static word representations with contextualized semantic information while remaining an input-embedding alternative to contextualized representations.
Abstract
from arXiv · showhide
GloVe learns word embeddings by leveraging statistical information from word co-occurrence matrices. However, word pairs in the matrices are extracted from a predefined local context window, which might lead to limited word pairs and potentially semantic irrelevant word pairs. In this paper, we propose SemGloVe, which distills semantic co-occurrences from BERT into static GloVe word embeddings. Particularly, we propose two models to extract co-occurrence statistics based on either the masked language model or the multi-head attention weights of BERT. Our methods can extract word pairs without limiting by the local window assumption and can define the co-occurrence weights by directly considering the semantic distance between word pairs. Experiments on several word similarity datasets and four external tasks show that SemGloVe can outperform GloVe.
1 Introduction
SemGloVe distills semantic co-occurrences from BERT into static GloVe embeddings to address local-window limits and position-based relevance weighting. Its attention- and masked-language-model-based variants outperform GloVe across intrinsic and extrinsic evaluations.
- Motivation: GloVe extracts word pairs within a predefined local window and weights them by position, which can omit relevant pairs and misrepresent semantic relevance.For example, the method may assign a lower score to a semantically related pair simply because the words are farther apart.
- Approach: SemGloVe distills contextualized semantic information from BERT into static word embeddings by replacing GloVe’s hard co-occurrence counts with semantic co-occurrences.The paper defines semantic co-occurrences using BERT-derived scores rather than only corpus proximity.
- Approach: The SAN model uses BERT self-attention weights to score word pairs, allowing semantic relevance to outweigh positional closeness.In the example, “king-queen” receives a higher score than “king-to” despite “to” being closer to “king.”
- Approach: The MLM model masks target words, predicts context words from the whole vocabulary, and uses output logits to construct co-occurrence statistics without a local-window restriction.This produces word pairs such as “king-queen” and “king-crown” from BERT’s predictions.
- Results: SemGloVe outperforms GloVe on word similarity datasets and external Chunking, POS tagging, and NER tasks, while identifying pairs and counts that are more semantically relevant.The paper also reports better averaged results than existing state-of-the-art noncontextualized embedding methods on intrinsic and extrinsic evaluations.
2 Related Work
Prior word-representation methods learn from local context, syntactic structure, subwords, or external knowledge bases. SemGloVe instead distills semantic information from large pretrained language models over unlabelled corpora.
- Word Representations: Skip-Gram and CBOW learn word vectors from local context, while dependency-based methods inject syntactic information from parse trees.FastText further enriches representations with subword information.
- Word Representations: Some prior methods combine text corpora with structured knowledge bases containing semantic relations, but constructing such resources is expensive.SemGloVe draws on large-scale pretrained language models rather than manually constructed structured data.
- Distilling Knowledge From BERT: Knowledge distillation transfers information from a large teacher network to a smaller student model, motivating approaches that distill BERT knowledge into compact representations.The paper positions SemGloVe within this broader distillation line of work.
3 Background
GloVe builds embeddings from a global word-word co-occurrence matrix whose entries use local position-based weighting and are optimized through a weighted factorization objective. BERT instead uses bidirectional Transformer layers and masked-token prediction to model contextual information.
- 3.1 GloVe: GloVe constructs a global matrix X whose entry Xij counts occurrences of context word wj around target word wi within a local window.The vocabulary V contains the corpus words, and nearer context positions receive larger weights.
- 3.1 GloVe: GloVe learns target and context embeddings by optimizing a weighted loss over matrix entries, with biases and a weighting function that downweights infrequent co-occurrences.The weighting function uses hyper-parameters xmax and α.
- 3.1 GloVe: After optimization, GloVe combines the target and context vectors to obtain the final embedding for each word.The supplied passages describe the final representation as ei + e′i.
- 3.2 BERT: BERT is trained with masked language modeling over raw text using multiple self-attention encoder layers in a deep bidirectional Transformer.A token is masked and predicted from the contextualized output of the topmost layer.
- 3.2 BERT: For a masked token sequence, BERT adds position embeddings to token representations and repeatedly applies SAN encoder layers before predicting the masked word.The model contains K SAN layers, with Wp denoting the position-embedding matrix.
4 Semantic GloVe
SemGloVe replaces GloVe’s hard, position-based co-occurrence counts with semantic values distilled from BERT through self-attention or masked-language-model predictions. The resulting SemGloVesd and SemGloVemd models construct global word-word statistics by selecting semantically related contexts and aggregating them across the corpus.
- 4.1 Semantic Co-occurrences from Multi-Head Self-Attention: SemGloVesd uses BERT’s multi-head self-attention weights to measure semantic relationships instead of GloVe’s heuristic position-based distance.Attention weights are aggregated across layers and heads, converted from BPE-to-BPE to word-to-word values, and used to form semantic co-occurrence counts.
- 4.1 Semantic Co-occurrences from Multi-Head Self-Attention: The self-attention method averages BPE-level attention weights into word-level values, retains the top-S context words, and applies the Division distance function before corpus-wide accumulation.The procedure first constructs a word-to-word attention matrix, filters semantic-irrelevant words by descending attention weight, and accumulates the resulting counts globally.
- 4.2 Semantic Co-occurrences from Masked Language Model: The MLM method selects the top 2S predicted BPE tokens by logits, computes token-level semantic distances, accumulates global BPE co-occurrences, and averages them into word-level counts.This dynamic context construction uses BERT’s deep bidirectional Transformer outputs rather than relying only on locally observed word pairs.
- 4.3 Experiments: Figure 2 evaluates WS353S performance as vector dimension, corpus size, and xmax vary.The figure organizes the sensitivity analysis across these three settings, while the supplied caption does not state specific plotted outcomes.
- 4.2 Semantic Co-occurrences from Masked Language Model: SemGloVemd is the GloVe model trained on semantic co-occurrences derived from BERT’s masked language model.The method defines context from MLM output tokens, ranks them by logits, and converts BPE-level counts into a word-word co-occurrence matrix.
5 Experiments
SemGloVe is evaluated against GloVe and other embedding baselines on intrinsic similarity, downstream sequence-labeling, and targeted analysis tasks. Across these evaluations, its masked-language-model variant delivers the strongest overall gains, while analyses examine distance functions, context size, corpus size, and semantic co-occurrences.
- 5.3 Development Experiments: SemGloVe outperformed GloVe across vector dimensions, corpus sizes, and xmax settings, while SemGloVemd remained strong with small corpora and stable across xmax values.Both methods improved as corpus size increased, but the MLM-based variant retained reasonable performance with limited data; 300 dimensions was selected for final experiments.
- 5.4 Final Results: 33.4% absolute average improvement over GloVe on four word-similarity datasets was achieved by SemGloVemd, whose averaged results also outperformed the best methods in the literature.SemGloVesd achieved a 5.6% absolute average increase, while SemGloVemd performed better and was selected as the final SemGloVe variant.
- 5.4 Final Results: 0.47% absolute average improvement over GloVe was obtained across Chunking, POS tagging, and NER, with SemGloVe also showing competitive performance against BERTbase.The comparison is intended to demonstrate that static SemGloVe vectors remain useful for downstream tasks, not that contextualized embeddings are weak.
- 5.5 Analysis: SemGloVemd generated semantic word pairs unavailable to local-window methods and produced clearer class separation than GloVe or SemGloVesd in t-SNE visualizations.The MLM variant can identify similar words over the whole vocabulary using deep bidirectional context, while subword-pair averaging enriches its embeddings.
- 5.5 Analysis: The Division distance function improved over GloVe’s position-based distance when mapping BERT weights into co-occurrence counts.The comparison replaces Division with the position-based function in SAN- and MLM-based variants to test how BERT weights are converted into co-occurrence counts.
- 5.5 Analysis: Reducing SAN-based context from ten to five words preserved valuable word pairs, whereas reducing the MLM-based context window decreased performance.The ten-word SAN setting remained better than GloVe but was slightly worse than the five-word setting.
6 Conclusion
SemGloVe replaces GloVe’s hard co-occurrence counts by distilling semantic co-occurrences from BERT, removing local-window constraints and incorporating semantic distance. Intrinsic and extrinsic experiments show that SemGloVe outperforms GloVe.
- SemGloVe replaces GloVe’s hard counts by distilling semantic co-occurrences from BERT into static word embeddings.
- The method extracts word pairs without local-window constraints and counts co-occurrences using semantic distance between word pairs.
- Intrinsic and extrinsic experiments show that SemGloVe outperforms GloVe.