Source-linked AI summary
Issues in evaluating semantic spaces using word analogies
Tal Linzen
TL;DR
Word analogy evaluation is intended to test whether semantic relations form consistent vector offsets, but cosine-based retrieval may also reward neighborhood structure. The paper introduces offset-free baselines and reversal tests, finding that analogy success often depends on these alternative signals rather than offset consistency alone.
Problem
Cosine-based analogy accuracy may evaluate both offset consistency and the neighborhood structure around the constructed point, weakening its interpretation as evidence for consistent offsets.
Method
The paper compares the offset method with baselines that ignore a or both a and a∗, and tests accuracy after reversing analogy directions.
Results
The offset method’s success was partly reproduced by baselines ignoring analogy inputs, and reversing direction substantially changed accuracy.
Takeaways & Limitations
Analogy evaluations should report simple baselines such as ONLY-B and IGNORE-A to distinguish offset evidence from neighborhood-driven performance.
Takeaways & Limitations
The conclusions are based on the analogy sets considered, and other offset-consistency methods may be less vulnerable to trivial responses and neighborhood structure.
Abstract
from arXiv · showhide
The offset method for solving word analogies has become a standard evaluation tool for vector-space semantic models: it is considered desirable for a space to represent semantic relations as consistent vector offsets. We show that the method's reliance on cosine similarity conflates offset consistency with largely irrelevant neighborhood structure, and propose simple baselines that should be used to improve the utility of the method in vector space evaluation.
1 Introduction
The offset method evaluates whether semantic relations are encoded as consistent vector offsets, but cosine-based nearest-neighbor retrieval may also reflect unrelated neighborhood structure. The paper proposes baselines and reversal tests to assess how much analogy accuracy evidences offset consistency.
- Motivation: VSM analogy evaluation tests whether a relation such as base-to-gerund is represented as a consistent linear offset across word pairs.For example, an offset learned from debug:debugging is applied to scream to predict screaming.
- Method: The method returns the vocabulary word with greatest cosine similarity to the point obtained by adding the source offset to the target word.Because VSMs are continuous, the computed landing point usually does not coincide exactly with a vocabulary word.
- Problem: Cosine-based retrieval may conflate offset consistency with neighborhood structure around the constructed point.When a∗ and a are similar, the nearest neighbor of a∗−a+b may simply be a nearest neighbor of b.
- Approach: The paper proposes baselines that omit the offset and tests whether reversing analogy direction changes accuracy.If the method measures offset consistency, reversing the direction should not affect accuracy.
2 Analogy functions
The paper compares the offset method with baselines that ignore some analogy words or reverse the task, exposing how retrieval choices can produce correct answers without demonstrating offset consistency.
- Setup: The experiments allow every vocabulary word as a guess, while noting that random-word performance is extremely low because vocabularies are large.The methods differ in which analogy words they exclude or use when selecting the answer.
- Offset methods: VANILLA applies the offset method literally, whereas ADD excludes a, a∗, and b because the unfiltered solution is often one of those words.The standard practical version therefore removes the three input words from the candidate pool.
- Baselines: ONLY-B ignores a and a∗ and returns the nearest neighbor of b, while IGNORE-A selects the word most similar to both a∗ and b.ONLY-B can succeed when a∗−a is small and b∗ is already near b; IGNORE-A identifies a point midway between a∗ and b.
- Baselines: ADD-OPPOSITE tests whether the expected answer remains recoverable even when the search moves opposite to the estimated offset.This extends the logic that sparse neighborhoods around b can yield correct answers without consistent offsets.
- Reversal: REVERSE (ADD) applies ADD to the reversed analogy, while REVERSE (ONLY-B) applies ONLY-B in that reversed direction.Direction-specific accuracy differences can indicate unequal neighborhood sparsity between the two word categories.
3 Experimental setup
The experiments evaluate analogy functions on a standard 14-category dataset across three skip-gram spaces that differ in context-window size. The analysis compares performance within space s5 and across spaces.
- Analogy problems: The analogy dataset contains 14 categories spanning morphological relations, world knowledge, and masculine–feminine concept pairs.Accuracy is calculated separately for each category.
- Semantic spaces: The study compares three skip-gram spaces trained on ukWaC and English Wikipedia, differing only in context-window parameters.The spaces are named s2, s5, and s10.
- Semantic spaces: Space s2 uses a two-word window on either side, while s5 and s10 use dynamic windows of five and ten words on either side.The s5 window expands when context words are excluded for low or high frequency.
- Evaluation: Figure 4 reports the accuracy of all analogy functions on space s5.The figure supports within-space comparison of the functions.
4 Results
Results show that analogy accuracy and apparent offset quality depend substantially on baselines, analogy direction, category, and semantic-space context windows. The findings indicate that neighborhood structure contributes to performance alongside offset consistency.
- Baselines: ADD accuracy in space s5 ranged from .13 to .90 across categories, while MULTIPLY held a mean accuracy advantage of .03 over ADD.Performance was poor for currencies and adjectives-to-adverbs but high for capitals and comparatives.
- Baselines: ADD and MULTIPLY always outperformed the baselines, but the margin varied widely across categories.In plurals, ONLY-B reached .70 and ADD-OPPOSITE achieved .45.
- Baselines: IGNORE-A outperformed ONLY-B in ten of 14 categories, while excluding a, a∗, and b was necessary for the offset method to avoid trivial nearest-neighbor answers.Without exclusion, b was the nearest neighbor in 93% of cases and a∗ in 5%.
- Reversed analogies: Reversing analogy direction reduced accuracy in most categories by a mean of −0.11, and changes in ADD correlated with ONLY-B changes at Pearson’s r = .72.This pattern suggests that the offset method jointly evaluates offset consistency and whether b∗ is nearest to b.
- Reversed analogies: The largest reversal decrease occurred for US cities, where accuracy fell from .69 to .17.The authors plausibly relate this drop to the many-to-one city-to-state mapping.
- Comparison across spaces: Overall ADD accuracy was similar across spaces, but category breakdowns differed: s2 reached .7 versus .44 for s10 on base-to-third-person, while s10 reached .68 versus .42 for s2 on US cities.Smaller windows favored some morphological categories, whereas s10 had advantages in some world-knowledge categories.
- Comparison across spaces: Overall accuracy can be misleading because analogy counts vary considerably across categories.The all-capitals category therefore contributes more to the aggregate than gender.
- Comparison across spaces: The overall advantage of ADD over baselines was highest for s2 and lowest for s10.In nationalities and common capitals, similar accuracy across spaces reflected more IGNORE-A performance in s10 than in s2.
5 Discussion
The analogy offset method can succeed for reasons unrelated to consistent vector offsets, including neighborhood structure and trivial candidate exclusion. Baseline comparisons, reversal tests, and finer-grained category analysis are therefore needed when interpreting analogy accuracy.
- The offset method’s success has been interpreted as evidence that semantic relations are represented as consistent vector offsets.
- A literal implementation failed because the nearest neighbor of a∗−a+b was almost always b or a∗.
- Some analogy accuracy was reproduced by baselines ignoring a or both a and a∗, while reversing the same offset substantially changed accuracy.
- Baseline performance varied across categories and spaces, with plural analogies performing highly even when both input words were ignored.The authors suggest plural tasks may measure singular–plural proximity rather than vector-offset encoding; space s2 showed the largest gains over baselines.
- Future analogy studies should report ONLY-B and possibly IGNORE-A baselines because alternative offset-consistency methods may be less vulnerable to trivial responses and neighborhood structure.
- Accuracy averaged across heterogeneous, unbalanced analogy sets makes spaces difficult to compare and can obscure category-specific strengths relevant to different downstream tasks.
- Heuristics excluding a, a∗, and b may improve solver performance while weakening its interpretation as evidence for semantic-space geometry.
- Combining category-specific strengths may guide development of new semantic spaces.