Source-linked AI summary
AutoExtend: Extending Word Embeddings to Embeddings for Synsets and Lexemes
Sascha Rothe, Hinrich Schütze
TL;DR
Existing word embeddings do not directly represent synsets and lexemes, while learning such representations from annotated corpora is costly. AutoExtend derives these embeddings from word embeddings and resource constraints, retaining the same vector space and avoiding an additional corpus. It achieves state-of-the-art word similarity and word sense disambiguation performance, including an improvement of more than 1% in WSD performance.
Problem
Word embeddings do not directly cover non-word resource types such as WordNet synsets and lexemes, and annotated-corpus training requires expensive sense labeling.
Method
AutoExtend extends arbitrary word embeddings to synset and lexeme embeddings by formalizing resource constraints, using a computationally efficient diagonal formulation.
Results
AutoExtend achieves state-of-the-art performance on word similarity and word sense disambiguation, with more than 1% improvement in WSD performance.
Takeaways & Limitations
The resulting synset and lexeme embeddings remain in the word-embedding space and can extend resources beyond WordNet, including Freebase.
Takeaways & Limitations
The model assumes diagonal embedding-distribution matrices, an efficiency-oriented restriction that treats word embeddings independently by dimension.
Abstract
from arXiv · showhide
We present \textit{AutoExtend}, a system to learn embeddings for synsets and lexemes. It is flexible in that it can take any word embeddings as input and does not need an additional training corpus. The synset/lexeme embeddings obtained live in the same vector space as the word embeddings. A sparse tensor formalization guarantees efficiency and parallelizability. We use WordNet as a lexical resource, but AutoExtend can be easily applied to other resources like Freebase. AutoExtend achieves state-of-the-art performance on word similarity and word sense disambiguation tasks.
1 Introduction
AutoExtend extends word embeddings to synsets and lexemes by formalizing resource constraints, separating embedding learning from resource-specific extension. The approach targets useful non-word representations without requiring sense-annotated corpora and is designed for broad resource applicability.
- 1 Introduction: Synsets group interchangeable synonyms, while lexemes pair a word spelling with a particular meaning.
- 1 Introduction: Decoupling embedding learning from resource extension allows embeddings produced by improved learning methods to be used directly with resources.
- 1 Introduction: The method avoids training on additional sense-annotated corpora, whose required scale and labeling cost make them impractical.
- 1 Introduction: AutoExtend is presented as broadly applicable beyond WordNet, including to resources such as Wikipedia and Freebase.
- 1 Introduction: AutoExtend extends word embeddings to synset and lexeme embeddings using constraints encoded by resources such as WordNet.Words and synsets are modeled as sums of their lexemes, providing the basis for the formalization.
- 1 Introduction: The paper reports state-of-the-art word similarity and word sense disambiguation performance and releases implementation materials.
2 Model
AutoExtend extends word embeddings to lexeme and synset embeddings in the same space by formalizing WordNet relationships as sparse, dimension-independent constraints. An autoencoder learns these representations while incorporating bidirectional lexeme alignment and WordNet relation constraints.
- The model seeks word, lexeme, and synset embeddings that occupy the same embedding space.
- Words and synsets are modeled as sums of their constituent lexemes.For example, a word can combine multiple senses, while a synset combines lexemes from different words.
- Diagonal constraint matrices distribute embedding activations independently across dimensions, reducing parameters and supporting parallel computation.The matrices may assign different, including negative, weights to each dimension.
- 2.1 Learning: An autoencoder encodes word embeddings into synset embeddings and decodes synsets back into words using sparse rank-4 tensors.The hidden layer represents synsets, while lexeme representations arise on both encoding and decoding paths.
- 2.3 Lexeme embeddings: The method aligns the two lexeme representations and averages them in experiments, while WordNet relation constraints encourage related synsets to have similar embeddings.Relation constraints are especially useful for synsets containing only one lexeme.
- 2.4 WN relations: Training minimizes weighted synset, lexeme, and relation constraint errors with backpropagation under normalization and sparsity constraints.Because the constrained objective is not analytically solvable, the implementation uses gradient descent and exploits zero entries for efficiency.
3 Data, experiments and evaluation
The evaluation uses word2vec embeddings extended to lexemes and synsets, testing cross-type neighborhoods, WSD features, and contextual similarity. AutoExtend improves WSD by more than 1% and lexeme embeddings outperform previous similarity methods.
- Data and setup: 300-dimensional word2vec embeddings trained on approximately 10^11 tokens provide the input vectors for AutoExtend.The vocabulary contains 3,000,000 words and phrases, with coverage gaps between word2vec and WordNet.
- Data and setup: AutoExtend produces compatible word, lexeme, and synset embeddings whose nearest neighbors can be compared across all three data types.The experiment sets α = β = 0.33 because suitable weights for the three constraints are unknown.
- Word sense disambiguation: WSD experiments use Senseval-2 and Senseval-3 with IMS, a linear-SVM system implementing standard and AutoExtend-based feature sets.The AutoExtend features use synset vectors and a sentence centroid formed by summing non-stop-word word2vec vectors.
- Word sense disambiguation: S-product is the strongest standalone synset feature set, using element-wise products between sentence-centroid and synset-vector dimensions.S-cosine instead uses k cosine similarities, while S-raw concatenates centroid and synset-vector elements.
- Word sense disambiguation: 65.2% and 72.3% are the standard IMS accuracies on Senseval-2 and Senseval-3, respectively, while adding S-product yields the best reported configuration.The optimized combination uses α = 0.2 and β = 0.5, with only a small improvement over the strongest configuration described.
- Similarity evaluation: More than 1% improvement in WSD performance is achieved when AutoExtend features are used.For similarity evaluation, SCWS provides words, contexts, and human similarity scores; AutoExtend lexeme embeddings outperform previous work and synset embeddings.
4 Analysis
AutoExtend’s constraint weighting affects WSD and similarity differently, but maximum performance consistently requires combining synset, lexeme, and WordNet relation constraints.
- WSD-alone performs best with high weights on WordNet relations, while additional synset and lexeme weighting does not help.The best weightings lie at the top of the constraint triangle.
- WSD-additional benefits from nonzero weights for all three constraints, whose interaction with standard WSD features increases performance.The strongest region lies in the middle of the weighting triangle.
- SCWS performs best with limited WordNet-relation weighting because these relations bring related but dissimilar lexemes together.Its maximum lies in the lower part of the weighting triangle.
- AutoExtend never reaches maximum performance using only one constraint set; synset, lexeme, and WordNet relation constraints matter with application-specific weights.
- Using all four WordNet relation types outperforms any combination of three, although relation counts and affected parts of speech complicate comparisons.Relation types with more relations generally outperform those with fewer relations.
- Simple averaging is used for lexeme embeddings, and performance changes only slightly when the weighting parameter ranges from 0.2 to 0.8.
5 Resources other than WordNet
AutoExtend can extend embeddings beyond WordNet by substituting other resource entities and incorporating their aliases and relations, while also supporting multilingual alignment.
- For Freebase, synsets can be replaced by entities, aliases can act as associated words, and Freebase types can impose relations among entity embeddings.Examples include aliases for Barack Obama and similarity constraints for entities of the type “President of the US”.
- Multilingual extension requires word embeddings from different languages to share a vector space, achieved by learning a transformation matrix from known translations.Rows of the two matrices represent translation-equivalent words.
- Matrix inversion provides a faster way than gradient descent to learn the cross-lingual transformation matrix.
- The learned matrix transforms unknown embeddings into the new vector space, enabling synset embeddings from multilingual resources such as BabelNet.It also supports adding cross-linguistic relationships between synsets representing the same concept.
6 Related Work
Related work includes single-prototype, sense-specific, resource-based, and bilingual embedding methods; AutoExtend differs by extending existing embeddings without retraining them.
- Earlier distributed word-representation models produce one embedding per word, and these embeddings can serve as AutoExtend inputs.
- Sense-embedding methods derive representations from contexts, modify embedding objectives, or learn multiple prototypes, but their mappings to WordNet are not always clear.Some prior methods explicitly associate embeddings with WordNet synsets using glosses.
- Figure 3 compares different weightings of WordNet relation, lexeme, and synset constraints across WSD-additional, WSD-alone, and SCWS.The figure marks maxima with “x” and local minima with “o”.
- Unlike prior approaches that require a training step, AutoExtend can apply to any given word embeddings without retraining them.
- Other work tunes existing embeddings for task prediction, incorporates lexical knowledge during learning, or uses bilingual resources that require parallel data.
- SCWS is evaluated with contextual word similarity, while synset similarity is computed with cosine and can be contrasted with resource-only measures.
7 Conclusion
The paper presents AutoExtend as a general method for deriving synset and lexeme embeddings from word embeddings and reports state-of-the-art word similarity and WSD performance.
- AutoExtend learns synset and lexeme embeddings from word embeddings and generalizes to embedding sets and resources with suitable relationship constraints.The authors also release the extended embeddings and code for reproducing the WSD evaluation.