Source-linked AI summary
Take and Took, Gaggle and Goose, Book and Read: Evaluating the Utility of Vector Differences for Lexical Relation Learning
Ekaterina Vylomova, Laura Rimell, Trevor Cohn, Timothy Baldwin
TL;DR
Prior evaluations of vector subtraction focused on analogy completion and limited relation types, leaving its generality across lexical relations and learning settings open. This paper evaluates DIFFVECs with clustering and supervised classification, finding broad relation information and strong generalisation under suitable training, including to unseen lexical items.
Problem
Prior analogy-based evaluations covered limited lexical relation types and did not establish how broadly vector differences support different learning settings.
Method
The paper evaluates DIFFVECs using clustering and supervised classification across closed-world and open-world settings, including disjoint training and test vocabularies.
Results
DIFFVECs capture many morphosyntactic and morphosemantic relations, classify extremely well in closed-world settings, and improve substantially on open data with automatically generated negative samples.
Takeaways & Limitations
Vector differences encode lexical relations and, with suitable supervised training, generalise across a broad range of relations, including unseen lexical items.
Takeaways & Limitations
The evaluation constrains relations to binary, fixed-directional relations and words occurring in all embedding sets, excluding symmetric relations such as synonymy.
Abstract
from arXiv · showhide
Recent work on word embeddings has shown that simple vector subtraction over pre-trained embeddings is surprisingly effective at capturing different lexical relations, despite lacking explicit supervision. Prior work has evaluated this intriguing result using a word analogy prediction formulation and hand-selected relations, but the generality of the finding over a broader range of lexical relation types and different learning settings has not been evaluated. In this paper, we carry out such an evaluation in two learning settings: (1) spectral clustering to induce word relations, and (2) supervised learning to classify vector differences into relation types. We find that word embeddings capture a surprising amount of information, and that, under suitable supervised training, vector subtraction generalises well to a broad range of relations, including over unseen lexical items.
1 Introduction
This paper broadens evaluation of vector differences beyond analogy completion by testing lexical relation learning through clustering and classification. DIFFVECs capture many relation types, with clustering favoring syntactic relations and supervised classification succeeding in closed-world settings but requiring negative sampling for open-world data.
- Motivation: Analogy completion has shown that vector operations can encode relations such as opposite gender, capital-of, and pluralisation.Examples include king − man + woman ≈ queen, paris − france + poland ≈ warsaw, and cars − car + apple ≈ apples.
- Research gap: Prior analogy evaluations cover limited relation types and test only one-best answers, leaving the broader utility of DIFFVECs insufficiently examined.The paper addresses this gap with a larger dataset and unsupervised and supervised relation-learning tasks.
- Clustering: Clustering DIFFVECs works remarkably well, although syntactic relations are captured better than semantic ones.
- Classification: Classification achieves remarkably high accuracy in a closed-world setting but performs poorly when random, potentially unrelated word pairs are added.The open-world setting tests whether the classifier can distinguish related pairs from noise.
- Classification: Automatically synthesised negative instances substantially improve classification performance on open-world data.The approach aims to better align the learned class representation with the lexical relations.
- Embedding methods: Hyper-parameter optimised count-based methods are competitive with predict-based methods in both clustering and supervised relation classification.
2 Background and Related Work
Lexical relation learning spans extraction, classification, and relational similarity prediction, while vector-space methods have increasingly been used to represent such relations. Although vector differences have succeeded in analogy tasks, prior work had not systematically established which relation types they handle best or evaluated DIFFVECs directly through clustering and classification.
- Relation learning: Lexical relation learning includes extracting related word pairs, assigning predefined relation labels, and predicting relational similarity.
- Vector-space methods: Distributional word vectors have been applied to relations including hypernymy and qualia structure.
- Vector differences: Vector differences became prominent after word embeddings successfully modeled word analogy tasks.
- Research gap: Prior work analyzed why predict-based embeddings succeed, but no systematic study had identified the relation range where vector differences are most effective.
- Contribution: Compared with related work, this paper evaluates DIFFVECs through clustering and classification rather than only relational similarity.
3 General Approach and Resources
The paper evaluates DIFFVEC-based lexical relation learning across multiple pre-trained embeddings and a broad, directionally constrained relation dataset. It uses vector differences in clustering and supervised classification settings, with both closed-world and open-world formulations.
- Task definition: Lexical relation learning maps ordered word pairs to relation labels, either assuming every pair belongs to one relation or allowing a none-of-the-above label.The latter is the open-world setting, while the former is closed-world classification.
- DIFFVEC approach: The approach assumes that relation information is encoded in DIFFVECs, defined here as the difference w2 − w1 between word embeddings.The study focuses exclusively on vector subtraction rather than alternative vector-composition methods.
- Evaluation design: The study evaluates DIFFVECs through unsupervised clustering and supervised relation classification using selected embeddings and relation subsets.The supervised experiments use w2v, w2vwiki, and SVDwiki, chosen for superior clustering performance, and examine nine relations with sufficient training data.
- Word embeddings: The experiments compare five embedding models: w2v, GloVe, SENNA, HLBL, and count-based SVD.Wikipedia-trained versions of w2v, GloVe, and SVD are also included for consistency of comparison.
- Lexical relations: The dataset spans lexical semantic, morphosyntactic, and morphosemantic relations, restricted to binary relations with fixed directionality.Symmetric relations such as synonymy are excluded, and words must occur in all embedding sets.
- Lexical relations: The final dataset contains 12,458 relation-word triples covering 15 relation types from multiple lexical resources.Sources include SemEval’12, BLESS, the MSR analogy dataset, Princeton WordNet, Wiktionary, and additional lexical datasets.
4 Clustering
The clustering experiment tests whether DIFFVECs group word pairs by lexical relation, using spectral clustering over a 15-relation dataset. Results show meaningful but uneven structure: morphosyntactic relations cluster most cleanly, while lexical-semantic relations are harder.
- Method: Spectral clustering groups DIFFVECs and tunes cluster number and similarity measure by development-set V-Measure.The evaluation uses a 15-relation closed-world dataset and compares clustering against known lexical relation types.
- Overall results: w2v reaches a V-Measure of around 0.36, above GloVe and SVD at around 0.31 and HLBL and SENNA near 0.21.The w2v result remains relatively constant as the number of clusters varies; related word-sense-induction systems achieved under 0.2.
- Embedding sources: More training data improves clustering: w2vwiki and GloVewiki score below w2v and GloVe, while w2v has a clear empirical advantage over GloVe.SVDwiki performs almost as well as w2vwiki.
- Relation-level results: Morphosyntactic paradigm relations are easiest to capture, whereas lexical-semantic relations are hardest across embedding methods.The relation-level analysis uses entropy alongside V-Measure to assess clustering purity.
- Relation-level results: For w2v, one cluster contains around 90% of VERB3 pairs, with most errors caused by part-of-speech ambiguity.Ambiguous pairs such as (study, studies), (run, runs), and (like, likes) have above-average DIFFVEC distances and are grouped with cross-POS relations.
- Relation-level results: NOUNColl pairs form a large mixed cluster dominated by pairs whose second word denotes an animal, showing that one word’s lexical semantics can overwhelm the DIFFVEC.LEXSEMMero is instead split into topical clusters such as weapons, dwellings, and vehicles.
5 Classification
The paper evaluates supervised DIFFVEC classification in closed- and open-world settings, including tests with random pairs and disjoint vocabularies. Negative sampling makes classifiers more conservative, improving precision and overall F-scores in most open-world conditions while reducing recall.
- Classification settings: The study trains multiclass and binary classifiers over DIFFVECs for closed-world and open-world lexical-relation classification.The open-world evaluation includes random word pairs, while disjoint training and test vocabularies assess lexical memorisation.
- 5.1 CLOSED-WORLD Classification: A linear SVM achieves higher F-score than clustering with majority-class labelling on almost every closed-world relation.Most relations, including difficult clustering cases, receive very high F-scores; PREFIX has lower recall because word usages differ substantially.
- 5.2 OPEN-WORLD Classification: Open-world classification exposes poor performance when random word pairs are included, because many pairs do not correspond to training relations.The evaluation is designed to test whether classifiers can distinguish related pairs from noise.
- 5.3 OPEN-WORLD Training with Negative Sampling: Negative distractors produce higher precision and somewhat lower recall by encouraging more conservative predictions.Opposite pairs target relation asymmetry, while shuffled pairs prevent reliance on word-class properties; both are added in equal numbers to valid relations.
- 5.3 OPEN-WORLD Training with Negative Sampling: Negative sampling yields higher F-scores than standard training for most relations, except hypernyms and prefixes.The approach substantially reduces false positives, including false-positive noun pairs for LEXSEMMero.
- 5.4 Lexical Memorisation: With disjoint training and test vocabularies, negative sampling maintains precision as random pairs increase, though recall is much lower.At the maximum random-pair level, the negative-sampling classifier has higher F-score than the standard classifier.
6 Conclusions
The study finds that vector differences capture many lexical relations, with strongest utility under supervised classification. Clustering captures morphosyntactic and morphosemantic relations better than lexical semantic relations, while negative sampling improves classification beyond closed-world settings.
- DIFFVECs capture many morphosyntactic and morphosemantic relations, but lexical semantic relations are captured less well by clustering.
- Classification over DIFFVECs works extremely well in a closed-world setting, showing that their dimensions encode lexical relations.
- Classification performs less well on open data, but automatically generated negative samples substantially improve results.
- Negative sampling also improves classification when training and test vocabularies are split to reduce lexical memorisation.
- Overall, DIFFVECs have impressive utility across a broad range of lexical relations, especially under supervised classification.