Source-linked AI summary
Specialising Word Vectors for Lexical Entailment
Ivan Vulić, Nikola Mrkšić
TL;DR
Distributional spaces conflate word relationships, while prior specialisation work chiefly targets similarity and relatedness rather than lexical entailment. LEAR post-processes vectors by combining norm-based hierarchy with cosine similarity. It reports new highs across lexical-entailment tasks and a HyperLex Spearman increase from 0.540 to 0.686.
Problem
Distributional spaces conflate synonymy and topical relatedness, while prior specialisation chiefly targets similarity versus relatedness rather than lexical entailment, which organizes semantic networks hierarchically.
Source: §1 · p. 1Method
LEAR post-processes vectors with directed lexical-entailment constraints, combining an asymmetric norm-based hierarchy signal with symmetric cosine similarity.
Source: §1 · p. 2, §2.2 · p. 4Results
Takeaways & Limitations
Jointly using both cost terms yields the strongest performance across lexical-entailment tasks while cosine-based semantic-similarity results remain similar to ATTRACT-REPEL.
Source: §4.3 · p. 8The WORD2GAUSS comparison on HyperLex is not strictly fair because WORD2GAUSS does not leverage external knowledge.
Source: §4.2 · p. 7
Abstract
from arXiv · showhide
We present LEAR (Lexical Entailment Attract-Repel), a novel post-processing method that transforms any input word vector space to emphasise the asymmetric relation of lexical entailment (LE), also known as the IS-A or hyponymy-hypernymy relation. By injecting external linguistic constraints (e.g., WordNet links) into the initial vector space, the LE specialisation procedure brings true hyponymy-hypernymy pairs closer together in the transformed Euclidean space. The proposed asymmetric distance measure adjusts the norms of word vectors to reflect the actual WordNet-style hierarchy of concepts. Simultaneously, a joint objective enforces semantic similarity using the symmetric cosine distance, yielding a vector space specialised for both lexical relations at once. LEAR specialisation achieves state-of-the-art performance in the tasks of hypernymy directionality, hypernymy detection, and graded lexical entailment, demonstrating the effectiveness and robustness of the proposed asymmetric specialisation model.
1 Introduction
The paper introduces LEAR, a post-processing model for lexical entailment that addresses distributional spaces’ conflation of word relationships. It combines cosine-based similarity with norm-encoded hierarchy and reports gains across lexical-entailment tasks.
1 Introduction
- Distributional embeddings conflate synonymy and topical relatedness, while prior specialisation work predominantly distinguished similarity from relatedness rather than modeling lexical entailment.Source: §1 · p. 1
- Post-processing uses external linguistic constraints while treating the input vector space as a black box, making the approach applicable to any input space.Source: §1 · p. 1
- Lexical entailment organizes semantic networks hierarchically and supports tasks including taxonomy creation, natural-language inference, text generation, and metaphor detection.Source: §1 · p. 1
- LEAR (Lexical Entailment Attract-Repel) is the proposed model for specialising vector spaces for lexical entailment, building on an attract-repel framework.Source: §1 · p. 1
- LEAR jointly pulls constrained pairs together or pushes them apart, using cosine distance for similarity and vector norms to assign larger values to higher-level concepts.The intended ordering is |animal| > |dog| > |terrier|.Source: §1 · p. 2
- LEAR improved over the strongest baselines across three word-level lexical-entailment tasks and raised HyperLex Spearman correlation from 0.540 to 0.686 on 2,616 pairs.The improvements held across each of four input spaces.Source: §1 · p. 2
2 Methodology
ATTRACT-REPEL uses in-batch negative examples to pull ATTRACT pairs together, push REPEL pairs apart, and preserve input-vector information. LEAR adds directed lexical-entailment constraints, encoding hierarchy asymmetrically while jointly preserving semantic similarity.
2.1 The ATTRACT-REPEL Framework
- In ATTRACT-REPEL, in-batch negative examples make ATTRACT pairs closer to each other and REPEL pairs farther apart relative to their respective negatives.For ATTRACT pairs, negatives are the closest in-batch vectors to each member; for REPEL pairs, they are the furthest remaining in-batch vectors.Source: §2.1 · p. 2
- The ATTRACT term pulls constrained pairs together, while the REPEL term pushes constrained pairs apart.Their in-batch negatives set the relative-distance comparisons for each term.Source: §2.1 · pp. 2–3
- A regularisation term preserves high-quality semantic information in the distributional vectors when it does not contradict injected linguistic constraints.The regulariser penalises L2 distance from each word’s original distributional vector.Source: §2.1 · p. 3
2.2 LEAR: Encoding Lexical Entailment
- LEAR extends ATTRACT-REPEL with directed lexical-entailment constraints whose left-hand concepts are lower-level and right-hand concepts higher-level.The combined objective uses these constraints in both the asymmetric LE cost and the symmetric ATTRACT term.Source: §2.2 · p. 3
- Three norm-based asymmetric distances encode hierarchy by ordering higher- and lower-level concepts; two use norm ratios to avoid the unbounded incentive of the baseline distance.The ratio-based alternatives divide norm differences by either the sum or the larger norm.Source: §2.2 · p. 3
- LEAR combines a symmetric ATTRACT term with an asymmetric lexical-entailment cost, so directed entailment pairs are modelled through both cosine similarity and asymmetric distance.Its decoding metric likewise combines the symmetric and asymmetric costs; the paper reports that both contribute to stronger performance across evaluated LE tasks.Source: §2.2 · pp. 3–4
3 Experimental Setup
The experiments apply LEAR to four publicly available English vector spaces and use lexical-entailment, synonymy, and antonymy constraints. WordNet entailment constraints include direct and transitive noun-noun and verb-verb pairs.
Starting Distributional Vectors
- LEAR is tested with four input spaces: SGNS, GLOVE Common Crawl, CONTEXT2VEC, and FASTTEXT.The vectors are 300-dimensional except CONTEXT2VEC, which is 600-dimensional; dependency-based SGNS showed similar result patterns.Source: §3 · p. 4
Linguistic Constraints
- The constraints cover directed lexical entailment, synonymy, and antonymy, using synonymy as ATTRACT pairs and antonymy as REPEL pairs.The synonymy and antonymy pairs come from WordNet and Roget.Source: §3 · p. 4
- WordNet supplies 1,545,630 lexical-entailment constraints, including direct and transitive noun-noun and verb-verb pairs.Other part-of-speech pairings are discarded.Source: §3 · p. 4
Training Setup
- Training uses the original ATTRACT-REPEL setup without fine-tuning, for five AdaGrad epochs with batch sizes of 128 for each pair type.The settings are δatt = 0.6, δrep = 0.0, and λreg = 10^-9.Source: §3 · p. 4
4 Results and Discussion
LEAR improves performance across word-level lexical-entailment tasks and graded HyperLex judgments. The evaluations also examine how norm-based distance variants, combined objectives, and input vector spaces affect performance.
4.1 LE Directionality and Detection
- 0.92 → 0.96 on directionality and 0.81 → 0.88 on detection plus directionality; LEAR outperformed all baselines across all three tasks.Improvements were smallest on the easiest task and largest on the most difficult task.Source: §4.1 · p. 5
- D2 and D3, which normalise norm differences, outperform D1’s unbounded norm distance, with the gap more pronounced on graded lexical entailment.The text attributes D1’s weakness to unbounded norms diminishing the symmetric cosine term’s importance.Source: §4.1 · pp. 5–6
- LEAR performed well across four input vector collections, and its gains over a WordNet lookup baseline suggest generalisation beyond the supplied pairwise constraints.The lookup baseline scored 0.82 on directionality and 0.80 on detection; LEAR scored 0.96 and 0.92 on those tasks using the same WordNet constraints.Source: §4.1 · p. 6
4.2 Graded Lexical Entailment
- 0.540 on HyperLex-All and 0.512 on HyperLex-Nouns were the prior highest scores reported for LE-focused representation models.These are prior-work results, not LEAR’s scores.Source: §4.2 · p. 6
- LEAR outperforms SLQS-SIM, VISUAL, ORDER-EMB, POINCARÉ, and HYPERVEC on graded HyperLex evaluation.The authors describe SLQS-SIM and VISUAL’s generality measures as too coarse-grained for graded judgments; the WORD2GAUSS comparison is not strictly fair because it lacks external knowledge.Source: §4.2 · p. 7
4.3 Further Discussion
- The FULL LEAR model, combining symmetric and asymmetric terms, yields the strongest performance across all evaluated lexical-entailment tasks.The comparison includes SYM-ONLY and ASYM-ONLY variants; Table 2 reports WBLESS, BLESS, HyperLex-All, and HyperLex-Nouns results for FASTTEXT with D2.Source: Table 2 · p. 8, §4.3 · p. 8
- LEAR’s asymmetric lexical-entailment term does not significantly change cosine-based semantic-similarity performance relative to ATTRACT-REPEL.Both models yield approximately 0.71 Spearman correlation on SimLex and 0.70 on SimVerb.Source: §4.3 · p. 8
5 Related Work
Earlier work either embeds lexical entailment directly or uses vectors in supervised detection, while LEAR post-processes existing spaces for asymmetric hierarchy. The paper contrasts this flexible Euclidean approach with WordNet-based models that have narrower task or model constraints.
5 Related Work
- Earlier embedding approaches either learn lexical entailment directly in LE-focused spaces or use vectors as features for supervised detection models.The cited approaches include Yu et al. (2015), Vendrov et al. (2016), and Shwartz et al. (2016).Source: §5 · pp. 8–9
- Yu et al. (2015) and Tuan et al. (2016) use WordNet constraints, but cannot make directionality judgements; Nguyen et al. (2017) combines WordNet with SGNS.Because HYPERVEC is tied to SGNS, changing the distributional modelling paradigm requires changing the model.Source: §5 · p. 9
- LEAR-specialised vectors achieve substantially better performance across all evaluated lexical-entailment tasks than prior models discussed here.The passage also notes that HYPERVEC is tied to the SGNS objective, making it less versatile than LEAR.Source: §5 · p. 9
- ORDER-EMB encodes hierarchy through per-coordinate ordering, while POINCARÉ uses hyperbolic spaces to represent hierarchy and semantic similarity.LEAR instead induces LE-specialised embeddings in Euclidean vector spaces.Source: §5 · p. 9
6 Conclusion and Future Work
LEAR jointly specialises existing vector spaces for lexical entailment and semantic similarity, using vector norms to encode hierarchy. Its specialised vectors attain state-of-the-art semantic-similarity performance and new high scores across four lexical-entailment tasks; future work extends the approach to other relations and settings.
6 Conclusion and Future Work
- LEAR jointly injects symmetric and asymmetric constraints into existing vector spaces to specialise them for lexical entailment and semantic similarity.Its asymmetric cost encodes hierarchy through vector norms, assigning higher norms to higher-level concepts.Source: §6 · p. 9
- LEAR-specialised vectors attain state-of-the-art performance in semantic similarity and set new high scores across four lexical-entailment tasks.The paper describes the joint specialisation of both relations as having a synergistic effect.Source: §6 · p. 9
- Future work will test the method on other asymmetric relations, finer-grained hierarchy path lengths, unseen words, downstream NLU, and cross-lingual applications.The authors specifically mention meronymy and cross-lingual lexical entailment.Source: §6 · p. 9