PaperlayerView exact revision on arXiv

Source-linked AI summary

ConceptNet 5.5: An Open Multilingual Graph of General Knowledge

Robyn Speer, Joshua Chin, Catherine Havasi

arXiv:1612.03975v2cs.CL

TL;DR

Word embeddings lack some relational and general knowledge useful for understanding language. This paper presents ConceptNet 5.5 and ConceptNet Numberbatch, a hybrid of ConceptNet and distributional semantics, achieving state-of-the-art word-relatedness results and 56.1% SAT-analogy accuracy.

  • Problem

    Word-embedding methods can benefit from external relational and general knowledge that distributional semantics alone does not represent.

  • Method

    The paper releases multilingual ConceptNet 5.5 and combines its graph knowledge with distributional embeddings through a generalized retrofitting method called ConceptNet Numberbatch.

  • Results

    ConceptNet Numberbatch achieves state-of-the-art word-relatedness performance across multiple evaluations and 56.1% accuracy on SAT-style analogies, tying the previous best overall system.

  • Takeaways & Limitations

    Word-embedding applications can incorporate relational knowledge through ConceptNet, with the resource intended for use across many domains and languages.

  • Takeaways & Limitations

    The bag-of-vectors story model captures topical relatedness but does not represent how one event sensibly follows another, so its score is not actual story understanding.

Abstract

from arXiv · show

Machine learning about language can be improved by supplying it with specific knowledge and sources of external information. We present here a new version of the linked open data resource ConceptNet that is particularly well suited to be used with modern NLP techniques such as word embeddings. ConceptNet is a knowledge graph that connects words and phrases of natural language with labeled edges. Its knowledge is collected from many sources that include expert-created resources, crowd-sourcing, and games with a purpose. It is designed to represent the general knowledge involved in understanding language, improving natural language applications by allowing the application to better understand the meanings behind the words people use. When ConceptNet is combined with word embeddings acquired from distributional semantics (such as word2vec), it provides applications with understanding that they would not acquire from distributional semantics alone, nor from narrower resources such as WordNet or DBPedia. We demonstrate this with state-of-the-art results on intrinsic evaluations of word relatedness that translate into improvements on applications of word vectors, including solving SAT-style analogies.

Introduction

ConceptNet 5.5 is a multilingual knowledge graph expanded with lexical and world knowledge from many sources, designed to improve NLP systems using word embeddings. Its hybrid ConceptNet Numberbatch semantic space combines distributional semantics with ConceptNet and improves word-relatedness evaluations and SAT-style analogy performance.

  • ConceptNet 5.5: ConceptNet 5.5 connects natural-language words and phrases with labeled, weighted edges and incorporates lexical and world knowledge from many sources in many languages.It also links its knowledge to external resources including WordNet, Wiktionary, OpenCyc, and DBPedia.
  • Motivation: ConceptNet’s graph-structured knowledge can make semantic spaces more effective than distributional semantics alone, particularly for word-embedding-based NLP algorithms.The paper presents word embeddings as one application with clear benefits and compatibility with ongoing distributional-semantics research.
  • ConceptNet Numberbatch: ConceptNet Numberbatch combines distributional semantics and ConceptNet through a generalized retrofitting method as a separate hybrid semantic-space artifact.The authors identify this combination as the most effective semantic space.
  • Results: ConceptNet Numberbatch significantly outperforms other systems across many word-relatedness evaluations, with gains that translate to downstream improvements such as analogies.Its SAT-style analogy accuracy is 56.1%, outperforming other word-embedding systems and tying Turney’s LRA, the previous best overall system.

Related Work

ConceptNet is positioned among general-knowledge resources as a sufficiently large, freely available graph focused on common-sense meanings of words rather than named entities. Related embedding approaches include distributional word vectors, downloadable word2vec and GloVe matrices, and knowledge-graph-trained holographic embeddings.

  • General-knowledge resources: Cyc, DBPedia, and Google Knowledge Graph are prominent general-knowledge resources, differing in representation, source, focus, or availability.Cyc uses predicate logic; DBPedia extracts Wikipedia infobox facts focused largely on named entities, while Google Knowledge Graph is not freely available.
  • ConceptNet’s position: ConceptNet provides a sufficiently large, free knowledge graph focused on common-sense meanings of words in natural language rather than named entities.Its focus on words makes it particularly compatible with representing word meanings as vectors.
  • Word embeddings: Word embeddings represent words as dense unit vectors whose proximity captures semantic relatedness, often learned by predicting a word from neighboring words.This approach represents meaning as a continuous space where similarity and relatedness can be treated as a metric.
  • Word embeddings: word2vec and GloVe 1.2 are downloadable embedding matrices used as comparison points and ensemble inputs.word2vec was trained on 100 billion Google News words, while GloVe 1.2 was trained on 840 billion Common Crawl words.
  • Knowledge-graph embeddings: Holographic embeddings learn from labeled knowledge graphs by constraining circular correlation to represent relations, but ConceptNet experiments converged too slowly to evaluate.The authors suggest optimization and additional computing power could eventually overcome this limitation.

Structure of ConceptNet

ConceptNet 5.5 integrates multilingual knowledge from diverse sources into a large graph of language-independent, labeled relations. It standardizes terms and preserves links to external resources while supporting ambiguous and more specific term representations.

  • Knowledge sources: ConceptNet combines OMCS and multilingual sister projects, parsed Wiktionary, purpose-built knowledge games, Open Multilingual WordNet, JMDict, OpenCyc, and a DBPedia subset.Wiktionary is its largest input source, providing 18.1 million edges and much of the graph’s multilingual vocabulary.
  • Relation structure: ConceptNet aligns its knowledge on 36 language-independent relations, using directed edges while designating relations such as SimilarTo as symmetric.Specific relations such as UsedFor and HasPrerequisite tend to connect common words and phrases, whereas rarer words use general relations such as Synonym and RelatedTo.
  • Term representation: Terms are Unicode-normalized in NFKC, lowercased, tokenized without punctuation, joined with underscores, and represented with language-coded /c URIs.Relations use a separate /r namespace and the same English relation names apply across languages.
  • Term representation: ConceptNet 5.5 removes ConceptNet 5.4’s English lemmatizer and instead relates word inflections while preserving their surface forms.The earlier representation conflated forms such as “drive” and “driving”; ConceptNet 5.5 changes how these terms are represented and connected.
  • Node and resource linking: Nodes usually represent undisambiguated words or phrases, with optional part-of-speech and further disambiguation linked through SenseOf, while ExternalURL preserves imported-data provenance and bidirectional Linked Open Data connections.The ambiguous term /c/en/lead can include the more specific noun-sense URI /c/en/lead/n.

Applying ConceptNet to Word Embeddings

The section derives ConceptNet-PPMI embeddings from a pruned graph matrix and combines ConceptNet with text-trained embeddings through expanded retrofitting. The resulting methods incorporate graph structure, multilingual connections, and complementary pretrained embedding sources.

  • ConceptNet-PPMI: ConceptNet is represented as a sparse, symmetric term-term matrix after pruning terms with fewer than three edges.Each matrix cell sums the weights of edges connecting the corresponding terms.
  • ConceptNet-PPMI: The matrix is converted into 300-dimensional ConceptNet-PPMI embeddings using smoothed PMI, PPMI clipping, truncated SVD, and symmetric term-context combination.These embeddings implicitly represent ConceptNet’s graph structure and approximate the connectedness of node pairs.
  • Expanded retrofitting: Expanded retrofitting adjusts pretrained vectors toward their ConceptNet neighbors, includes graph terms absent from the original vocabulary, and weights connections by ConceptNet edge strengths.For absent original values, the method effectively sets α_i = 0.
  • Expanded retrofitting: Expanded retrofitting exploits ConceptNet’s multilingual links to improve English representations through translations while embedding foreign-language terms in the same space.The method propagates information across multilingual connections in ConceptNet.
  • Merging embeddings: Retrofitting is applied to both word2vec and GloVe, whose aligned outputs are merged through a globally linear projection reduced to 300 dimensions.The alignment also infers compatible embeddings for terms missing from one vocabulary, and the merged vocabulary includes the pruned ConceptNet graph.

Evaluation

The evaluation compares ConceptNet Numberbatch 16.09 with other embedding systems on word relatedness and downstream semantic tasks. It uses held-out tests and qualifies multilingual and Story Cloze interpretations.

  • Systems evaluated: ConceptNet Numberbatch 16.09 is compared with word2vec SGNS, GloVe 1.2, LexVec, and ConceptNet-PPMI.Numberbatch is the hybrid of ConceptNet 5.5, word2vec, and GloVe.
  • Intrinsic relatedness: Intrinsic evaluation ranks word-pair relatedness and compares the resulting ordering with human judgments using Spearman correlation (ρ).Out-of-vocabulary pairs are assigned relatedness 0; evaluations include MEN-3000, RW, WordSim-353, and MTurk-771.
  • Evaluation protocol: Development and test splits are used to reduce manual overfitting, with MTurk-771 held out entirely as a final unseen evaluation.MEN-3000 uses its 1000-item test set, RW holds out one third, and WordSim-353 is used in development.
  • Multilingual evaluation: Spanish WordSim-353 illustrates non-English evaluation, but a thorough multilingual comparison is beyond scope because competing systems provide only English vectors.The paper attributes non-English vectors to expanded retrofitting despite English-only distributional semantics.
  • Proportional analogies: For ConceptNet Numberbatch 16.09, analogy scoring used w1 = 0.2 and w2 = 0.6, making transposed comparisons more important than directly stated comparisons.The score combines separate similarities, pair-difference dot products, and transposed pair-difference dot products.
  • Story Cloze Test: The Story Cloze model tests sensible story endings but uses only contextual word relatedness, so improved scores should not be portrayed as actual story understanding.The task is difficult for computers because it relies on implicit common-sense knowledge.

Results and Discussion

ConceptNet Numberbatch achieves state-of-the-art word-relatedness results and strong SAT-analogy performance, while Story Cloze performance remains modest. Combining distributional and relational knowledge produces more robust word embeddings that better match human judgments.

  • Word relatedness: ConceptNet Numberbatch 16.09 is state of the art on all four word-relatedness evaluations, including Rare Words, MEN-3000, and MTurk-771.Its performance exceeds other evaluated embeddings beyond the confidence interval of question selection, indicating high-confidence gains on both breadth and depth of understanding.
  • SAT-style analogies: 56.1% of SAT analogy questions were answered correctly by ConceptNet Numberbatch, including 58.8% on the held-out final-test half.This was the best result among word-embedding systems and matched or slightly exceeded best-in-class systems, although rankings could change because results overlapped within 95% confidence intervals.
  • SAT-style analogies: ConceptNet Numberbatch achieved SAT-analogy performance comparable to non-word-embedding systems despite using only vector addition and subtraction, without representing relationship types explicitly.The authors suggest incorporating ConceptNet’s relation labels could improve analogy performance further.
  • Story Cloze: ConceptNet Numberbatch selected the correct Story Cloze ending 59.4% of the time, slightly exceeding prior reported results but trailing LexVec’s 59.9% best score.The authors characterize this performance as acceptable but unremarkable and suggest bag-of-vectors methods may have reached a plateau near 59%.
  • Overall findings: Combining distributional semantics with relational knowledge outperformed either component alone, making word embeddings more robust and more correlated with human judgments.The authors recommend that embedding-based techniques include relational knowledge or use pre-trained embeddings that already incorporate it.

Availability of the Code and Data

ConceptNet 5.5’s code, documentation, knowledge graph, reproducible build process, evaluation graph, and evaluated Numberbatch embeddings are publicly available.

  • ConceptNet 5.5’s code and documentation are available on GitHub, while the knowledge graph can be browsed at conceptnet.io.The full build process and evaluation graph can be reproduced with the README’s Snakemake instructions, optionally using Docker Compose.
  • The September 2016 ConceptNet Numberbatch embeddings evaluated in the paper are downloadable as pre-built embeddings from the ConceptNet Numberbatch repository, tag 16.09.
Loading 1612.03975v2…