Source-linked AI summary
ConceptNet at SemEval-2017 Task 2: Extending Word Embeddings with Multilingual Relational Knowledge
Robyn Speer, Joanna Lowry-Duda
TL;DR
Multilingual word-similarity evaluation calls for representations that connect words across languages. This paper updates a system combining ConceptNet with distributional embeddings for SemEval-2017 Task 2, where it placed first in both subtasks. The results also support ConceptNet’s usefulness as multilingual glue for a shared representation.
Problem
The paper addresses multilingual word similarity with a system intended to represent relationships among words across languages.
Source: §1 · p. 1Method
Results
First place in both SemEval-2017 Task 2 subtasks; the system had the highest score on every test set except monolingual Farsi.
Source: §1 · p. 1, §3.4 · p. 4Takeaways & Limitations
The results transfer well to other languages, supporting ConceptNet’s usefulness as multilingual glue for a single representation.
Source: §4 · p. 4The system relies heavily on ConceptNet links encoding direct translations or exact cognates, and may struggle with words lacking known or accurate English translations.
Source: §4 · p. 4
Abstract
from arXiv · showhide
This paper describes Luminoso's participation in SemEval 2017 Task 2, "Multilingual and Cross-lingual Semantic Word Similarity", with a system based on ConceptNet. ConceptNet is an open, multilingual knowledge graph that focuses on general knowledge that relates the meanings of words and phrases. Our submission to SemEval was an update of previous work that builds high-quality, multilingual word embeddings from a combination of ConceptNet and distributional semantics. Our system took first place in both subtasks. It ranked first in 4 out of 5 of the separate languages, and also ranked first in all 10 of the cross-lingual language pairs.
1 Introduction
ConceptNet combines multilingual general knowledge with distributional semantics to create word embeddings. Luminoso updated this system for SemEval-2017 Task 2 and placed first in both subtasks.
1 Introduction
- ConceptNet Numberbatch combines ConceptNet with distributional semantics using an extension of retrofitting.The earlier system combined ConceptNet with word2vec Google News and GloVe embeddings to produce embeddings that performed strongly across word-relatedness evaluations.Source: §1 · p. 1
- The SemEval submission updated the system alongside ConceptNet 5.5.3 and added fallback methods for out-of-vocabulary words.The task also included comparisons across ten pairs of distinct languages.Source: §1 · p. 1
- First place in both subtasks: the system ranked word pairs for semantic similarity or relatedness within and across five languages.The languages were English, German, Italian, Spanish, and Farsi.Source: §1 · p. 1
2 Implementation
The implementation combines ConceptNet and distributional embeddings, selects a vocabulary, projects the combined vectors to 300 dimensions, and uses fallback strategies for out-of-vocabulary terms.
2 Implementation
- Expanded retrofitting adds knowledge-graph-only terms to the vocabulary and assigns them vectors, unlike ordinary retrofitting, which retains the input vocabulary.The resulting vectors are built from multiple embedding sources, aligned on a unified vocabulary, and reduced in dimensionality.Source: §2 · p. 1, §2.1 · p. 2
2.2 Vocabulary Selection
- The unified vocabulary contains 1,884,688 ConceptNet terms plus 99,869 additional terms selected from word2vec and GloVe.The selection uses high-frequency rows from those embeddings and their union with the ConceptNet subgraph terms.Source: §2.2 · p. 2
2.3 Dimensionality Reduction
- The 300-dimensional projection uses a 5% sample of the combined matrix and UΣ1/2 to reduce redundancy among concatenated features.The sample is assumed to represent approximately the same distribution as the full matrix.Source: §2.3 · p. 2
2.4 Don’t Take “OOV” for an Answer
- When a term is missing a vector, the system averages vectors of its ConceptNet neighbors; terms without such neighbors remain out of vocabulary.The task required a similarity value for every pair, so the submission added two further fallback strategies.Source: §2.4 · p. 3
- For non-English terms missing from the output matrix, the system first tries the vector for an identically spelled English term.This fallback is theoretically vulnerable to false cognates, though the authors say they tend to be common words and are unlikely to use it.Source: §2.4 · p. 3
- A second fallback strategy applies when a term cannot be found in its language or in English.The supplied passage introduces this strategy but does not describe its operation.Source: §2.4 · p. 3
3 Results
The submitted ConceptNet system ranked highest on both subtasks, while a parallel-text variant scored better in retrospect on released test data. The section describes the systems, their evaluation, and the comparison across languages.
3 Results
- Systems were scored using the harmonic mean of Pearson and Spearman correlations, aggregated over the top four languages in subtask 1 and top six language pairs in subtask 2.Source: §3 · p. 3
3.1 The Submitted System: ConceptNet + word2vec + GloVe
- The submitted system combined ConceptNet 5.5.3 with English word2vec and GloVe embeddings, with other-language vectors depending on ConceptNet’s multilingual links.Its aggregate scores were 0.743 on subtask 1 and 0.754 on subtask 2.Source: §3.1 · p. 3
3.2 Variant A: Adding Polyglot Embeddings
- Variant A added Polyglot embeddings for German, Spanish, Italian, and Farsi, but scored lower than the submitted system on trial and released test data.Its aggregate scores were .720 on subtask 1 and .736 on subtask 2.Source: §3.2 · pp. 3–4
3.3 Variant B: Adding Parallel Text from OpenSubtitles
- Variant B added 300-dimensional SVD-of-PPMI vectors from OpenSubtitles2016 cross-language and within-language co-occurrences; it was inconclusive on trial data but scored better than the submitted system on released test data.The authors submitted Run 2 because it was simpler and seemed slightly better on average in trials; Variant B’s released-test aggregate scores were .759 on subtask 1 and .767 on subtask 2.Source: §3.3 · p. 4
3.4 Comparison of Results
- Luminoso-Run2 had the highest aggregate score on both subtasks and the highest score on every test set except monolingual Farsi; Variant B performed best in the end.Table 1 compares the submitted system, its version without OOV strategies, Variants A and B, and the Nasari baseline.Source: §3.4 · p. 4
4 Discussion
The discussion attributes ConceptNet’s value to its relational knowledge and describes its multilingual transfer as useful. It also notes that reliance on translation links and cognates may limit performance on words that do not translate directly into English.
4 Discussion
- The authors say ConceptNet’s particular blend of crowdsourced, gamified, and expert knowledge yields a large evaluation-score improvement over distributional semantics alone.Source: §4 · p. 4
- Results transfer well across languages, supporting ConceptNet’s usefulness as multilingual glue for combining knowledge into a shared representation.Source: §4 · p. 4
- The submitted system relies heavily on translation links and exact cognates, and may struggle with words lacking known or accurate English translations; such vague or language-specific words tend not to appear in current similarity evaluations.For harder-to-translate words, the authors may need to rely more on distributional semantics from each language’s corpus, as in their unsubmitted variants.Source: §4 · p. 4