Source-linked AI summary
Integrating Distributional Lexical Contrast into Word Embeddings for Antonym-Synonym Distinction
Kim Anh Nguyen, Sabine Schulte im Walde, Ngoc Thang Vu
TL;DR
Distributional models often treat antonyms and synonyms as related because they occur in similar contexts, despite the need to distinguish these relations. The paper adds lexical contrast to feature weighting and skip-gram embeddings, and reports improvements on word similarity and antonym–synonym distinction.
Problem
Distributional models retrieve both synonyms and antonyms as related, although applications such as machine translation and textual entailment require distinguishing specific semantic relations.
Source: §1 · p. 1Method
The paper weights distributional features using lexical contrast and integrates contrast into a skip-gram objective to predict similarity and distinguish antonyms from synonyms.
Source: §1 · p. 2, §2.1 · p. 2, §2.2 · p. 3Results
The improved vectors significantly outperform original representations across word classes, while dLCE outperforms SGNS and mLCM on SimLex-999 similarity prediction and antonym–synonym distinction.
Source: §3.2 · p. 4, §3.3 · pp. 4–5Takeaways & Limitations
The reported results show that distinguishing strongly similar synonyms from often-related but dissimilar antonyms also supports distinguishing degrees of similarity.
Source: §3.3 · p. 5, §4 · p. 5
Abstract
from arXiv · showhide
We propose a novel vector representation that integrates lexical contrast into distributional vectors and strengthens the most salient features for determining degrees of word similarity. The improved vectors significantly outperform standard models and distinguish antonyms from synonyms with an average precision of 0.66-0.76 across word classes (adjectives, nouns, verbs). Moreover, we integrate the lexical contrast vectors into the objective function of a skip-gram model. The novel embedding outperforms state-of-the-art models on predicting word similarities in SimLex-999, and on distinguishing antonyms from synonyms.
1 Introduction
Antonymy and synonymy are important lexical relations, but distributional models often retrieve both as related because they occur in similar contexts. The paper uses lexical contrast in distributional vectors and word embeddings to distinguish them, building on prior distributional and embedding-based approaches.
1 Introduction
- Distinguishing the relations matters for machine translation and textual entailment, which require specific semantic relations rather than general relatedness.Source: §1 · p. 1
- Distributional models represent words through their co-occurring contexts, but often treat synonyms and antonyms alike as related words.The two relations can occur in similar contexts because words in them are interchangeable in some contexts.Source: §1 · p. 1
- Earlier approaches used distributional features, lexical resources, and word-embedding models to address antonym–synonym distinction.Examples include filtering distributionally similar words, using thesaurus contrasts, and integrating lexical resources into skip-gram models.Source: §1 · pp. 1–2
- Lexical contrast information is used in distributional vectors and word embeddings to distinguish antonyms from synonyms.The paper strengthens salient features and extends a skip-gram objective with lexical contrast.Source: §1 · p. 2
2 Our Approach
The approach combines a lexical-contrast feature weighting scheme with a skip-gram embedding objective that incorporates lexical contrast. The weighting compares target–synonym similarity with antonym–synonym similarity, while the embedding model applies contrast to target-word contexts.
2 Our Approach
- The section introduces two contributions: improved weighted features for antonym–synonym distinction and a skip-gram extension for similarity prediction and antonym identification.Source: §2 · p. 2
2.1 Improving the weights of feature vectors
- weightSA compares average target–synonym similarity with average similarity between the target’s antonyms and their synonyms.It uses cosine similarity and assigns zero when the target lacks lexical links or a feature does not co-occur with it.Source: §2.1 · p. 2
- The weighting scheme emphasizes features shared with synonyms but weaker among antonyms, while broadly shared features receive scores near zero.For formal, conception is positive, rumor negative, and issue near zero.Source: §2.1 · p. 2, Figure 1 · p. 3
- Because antonyms are fewer than synonyms, the method treats synonyms of an antonym as antonyms of the target.For example, bad and evil, and their synonyms, are used as antonyms for good.Source: §2.1 · pp. 2–3
2.2 Integrating the distributional lexical contrast into a skip-gram model
- The embedding approach adds lexical contrast to the SGNS objective, which models target–context co-occurrence and negative sampling.The added objective uses cosine similarity between embedded word vectors.Source: §2.2 · p. 3
- The dLCE objective integrates lexical contrast into skip-gram training by adding contrast across contexts of a target word.It applies lexical contrast to every context, unlike WE-TD and mLCM, which apply it to target words.Source: §2.2 · pp. 3–4
3 Experiments
The experiments evaluate lexical-contrast vectors and the dLCE embedding on antonym–synonym distinction and word similarity. The reported comparisons favor the improved vectors and dLCE over their respective baselines.
3.1 Experimental Settings
- The vector and embedding experiments use distributional information from a five-token window.The corpus is EN-COW14A, with approximately 14.5 billion tokens; WordNet and Wordnik supply lexical contrasts.Source: §3.1 · p. 4
3.2 Distinguishing antonyms from synonyms
- The first experiment evaluates weightSA on adjective, noun, and verb antonymous and synonymous pairs using AP.The gold standard contains 600 adjective pairs, 700 noun pairs, and 800 verb pairs.Source: §3.2 · p. 4
- Across word classes, weightSA significantly outperforms the original LMI representations, with and without SVD.Table 1 compares the representations using AP.Source: §3.2 · p. 4, Table 1 · p. 5
- Cosine similarities between antonymous and synonymous pairs differ more with weightSA than with LMI, and more still after SVD.Figure 2 compares the medians across word classes and four representation conditions.Source: §3.2 · p. 4, Figure 2 · p. 5
3.3 Effects of distributional lexical contrast on word embeddings
- The embedding experiments evaluate similarity with Spearman’s ρ and antonym–synonym distinction with AUC.The similarity evaluation compares ranked predictions with human ratings.Source: §3.3 · pp. 4–5
- On SimLex-999, dLCE outperforms SGNS and mLCM on predicting word similarity.The dataset was built to test similarity rather than relatedness or association.Source: §3.3 · p. 4, Table 2 · p. 5
- The improved antonym–synonym distinction in dLCE also supports distinguishing degrees of word similarity.The paper presents this as a consequence of better separating strongly similar synonyms from often strongly related but highly dissimilar antonyms.Source: §3.3 · p. 5
- On antonym–synonym distinction, dLCE outperforms SGNS and mLCM by AUC.The evaluation uses cosine similarities for word pairs from the dataset described in Section 3.2.Source: §3.3 · p. 5, Table 3 · p. 5
4 Conclusion
The paper proposes lexical-contrast representations for distributional semantics and word embeddings. It reports improved antonym–synonym distinction and prediction of word-similarity degrees.
4 Conclusion
- Lexical contrast improves weighted features for distinguishing antonyms from synonyms.The proposed vector representation applies to both traditional distributional semantics and word embeddings.Source: §4 · p. 5
- The paper incorporates lexical contrast into a skip-gram model to predict degrees of word similarity.The conclusion also reports that the model identifies antonyms.Source: §4 · p. 5
- The proposed vector representation enhances prediction of word similarity in distributional semantics and word embeddings.Source: §4 · p. 5