PaperlayerView exact revision on arXiv

Source-linked AI summary

Wikipedia-based Semantic Interpretation for Natural Language Processing

Evgeniy Gabrilovich, Shaul Markovitch

arXiv:1401.5697v1cs.CL

TL;DR

Natural-language semantics requires broad world knowledge, but prior approaches relied on statistical methods, limited lexical resources, or extensive manual encoding. ESA represents text in a high-dimensional space of Wikipedia concepts and improves text categorization and semantic-relatedness evaluation over prior state of the art.

  • Problem

    Natural-language semantic representation requires extensive common-sense and domain knowledge, while prior methods used statistical techniques, limited lexicons, or large manual knowledge-encoding efforts.

  • Method

    ESA represents text meaning as a high-dimensional space of explicit concepts corresponding to Wikipedia articles and their contents.

  • Results

    ESA produced significant improvements over prior state-of-the-art methods in text categorization and semantic relatedness, including correlations rising from r = 0.56 to 0.75 for words and from r = 0.60 to 0.72 for texts.

  • Takeaways & Limitations

    ESA provides a uniform, explainable concept-based representation for individual words and arbitrarily long text fragments while addressing synonymy and polysemy.

  • Takeaways & Limitations

    The short-document evaluation datasets were constructed by substituting document titles for classified full documents, which can misleadingly assign body-derived categories to titles.

Abstract

from arXiv · show

Adequate representation of natural language semantics requires access to vast amounts of common sense and domain-specific world knowledge. Prior work in the field was based on purely statistical techniques that did not make use of background knowledge, on limited lexicographic knowledge bases such as WordNet, or on huge manual efforts such as the CYC project. Here we propose a novel method, called Explicit Semantic Analysis (ESA), for fine-grained semantic interpretation of unrestricted natural language texts. Our method represents meaning in a high-dimensional space of concepts derived from Wikipedia, the largest encyclopedia in existence. We explicitly represent the meaning of any text in terms of Wikipedia-based concepts. We evaluate the effectiveness of our method on text categorization and on computing the degree of semantic relatedness between fragments of natural language text. Using ESA results in significant improvements over the previous state of the art in both tasks. Importantly, due to the use of natural concepts, the ESA model is easy to explain to human users.

1. Introduction

The paper introduces Explicit Semantic Analysis (ESA), which uses Wikipedia’s human knowledge to represent text meaning as explicit concepts for NLP tasks. It addresses limitations of statistical and lexicographic approaches while enabling applications such as categorization, similarity assessment, and external-knowledge interpretation.

  • Method: ESA represents natural-language meaning as a high-dimensional combination of Wikipedia concepts, using article contents as the knowledge associated with each concept.The method treats encyclopedia articles as concepts and compares input text with their associated articles to estimate concept relevance.
  • Motivation: Compared with bag-of-words and LSA, ESA incorporates large-scale world knowledge that cannot be deduced solely from the input text.The approach explicitly manipulates human-defined knowledge concepts rather than relying only on word occurrence patterns.
  • Motivation: ESA uses knowledge-rich concepts and their accompanying text to address synonymy, polysemy, and word-sense disambiguation.The concept representation moves beyond manipulating isolated words and uses article content to distinguish meanings.
  • Applications: The approach supports document-based NLP decisions, including information retrieval, text categorization, and comparing documents for similarity.Depending on the task, ESA uses full concept vectors or a smaller set of relevant concepts to enrich bag-of-words features.
  • Motivation: Wikipedia supplies external knowledge that helps interpret terse texts whose meaning depends on financial, technological, or other domain knowledge.For example, ESA connects “Bernanke takes charge” with Federal Reserve-related concepts and “Apple patents a Tablet Mac” with computing concepts.
  • Evaluation: ESA was evaluated on text categorization and semantic relatedness, producing significant improvements over existing state-of-the-art performance in both tasks.These evaluations test whether Wikipedia-based representations improve common NLP decisions and similarity judgments.

2. Explicit Semantic Analysis

Explicit Semantic Analysis represents text meaning as weighted combinations of Wikipedia concepts in a high-dimensional semantic space. It uses article text and links to estimate concept relevance and semantic relatedness.

  • Semantic representation: ESA represents each text fragment as a weighted vector over a canonical space of basic concepts.Each weight measures the association between the text and a concept; semantic relatedness can then be estimated from vector distance.
  • Wikipedia concepts: The concept set is intended to cover diverse topics, remain current, use human-recognized concepts, and associate each concept with explanatory text.These requirements support a broad, maintained semantic interpreter whose concepts can be matched against language.
  • Wikipedia concepts: Wikipedia articles serve as natural concepts, with each article’s text supporting affinity calculations between words and concepts.The method treats article titles as convenient references while treating articles themselves as atomic concepts.
  • Concept mapping: The word-by-concept table stores TFIDF-based term–concept associations and normalizes rows to reduce effects of document length.Its entries represent term relevance to concept documents, with logarithmic term frequency and document-frequency weighting.
  • Concept mapping: Text-fragment interpretations are formed by centering the vectors of their constituent words, enabling partial disambiguation of terms with multiple meanings.The examples of “mouse” and “screen” illustrate how contextual combinations can select different concept components.
  • Second-order interpretation: Second-order ESA incorporates knowledge from Wikipedia’s inter-article links, but linked concepts must be filtered because links do not always indicate strong relevance.The method distinguishes first-order article-text information from second-order link information and treats gratuitous links as a noise source.

3. Using Explicit Semantic Analysis for Computing Semantic Relatedness of Texts

ESA represents texts as vectors in a high-dimensional space of Wikipedia concepts, then computes semantic relatedness by comparing those vectors. Experiments compare Wikipedia-based interpretation with ODP and prior methods, including effects of knowledge breadth and second-order information.

  • 3.1 Automatic Computation of Semantic Relatedness: ESA represents each text fragment as an interpretation vector in a high-dimensional space of Wikipedia-derived concepts.The concepts correspond to Wikipedia articles, whose article bodies provide affinities between concepts and input words.
  • 3.1 Automatic Computation of Semantic Relatedness: Semantic relatedness is computed by comparing interpretation vectors, using cosine similarity throughout the reported experiments.
  • 3.1 Automatic Computation of Semantic Relatedness: Shared high-ranking Wikipedia concepts explain ESA’s similarity judgments, such as Scientific journal and Academic publication for related phrases.For “scientific article” and “journal publication,” these concepts appear among the top 20 concepts for each phrase.
  • 3.3.3 Results: Wikipedia-based interpretation outperforms ODP-based interpretation on both test collections, with statistically significant superiority for word relatedness at p < 0.005.The authors associate the difference with more independent concept axes and more focused, less noisy Wikipedia article text.
  • 3.2.2 The Effect of Knowledge Breadth: Using a newer Wikipedia snapshot improves results, although the difference between the two snapshot versions is small.The later snapshot increases the representation space from up to 171,332 to 241,393 features, with sparse vectors.
  • 3.3.3 Results: Second-order interpretation yields negligible improvements for semantic relatedness but improves text-categorization accuracy when only top-scoring concepts are retained.The authors attribute this difference to second-order information selectively adding related concepts in trimmed vectors.

4. Using Explicit Semantic Analysis for Text Categorization

The section presents ESA as a knowledge-based extension of text categorization, addressing limitations of bag-of-words representations by adding Wikipedia-derived features. It describes pruning and multi-resolution processing to make the representation usable for supervised classification.

  • Text categorization assigns documents one or more labels from a fixed category set for applications including routing, junk-email detection, and intelligence-report handling.
  • Bag-of-words systems are limited because they cannot generalize across vocabulary differences or use unseen testing words effectively.
  • 4.2 Using ESA for Feature Generation: ESA augments bag-of-words representations with knowledge-based features derived from Wikipedia concepts rather than replacing the original features.
  • 4.2 Using ESA for Feature Generation: Because using all Wikipedia concepts is computationally expensive, the categorization pipeline prunes interpretation vectors and retains selected high-scoring concepts.
  • 4.2 Using ESA for Feature Generation: Multi-resolution feature generation partitions documents into non-overlapping contexts, represents each with ESA, and pools the top ten concepts from all context vectors.
  • 4.3 Test Collections: Replacing full documents with titles creates misleading short-document labels because categories may depend on information present only in document bodies.

4.5 Wikipedia-based Feature Generation

Wikipedia-based feature generation enriches document representations with related concepts, including broader concepts from inter-article links. Qualitative examples and experiments show useful contextual features, improved categorization, and a limitation from spurious matches.

  • Feature Generation per se: Feature generation represents text fragments through Wikipedia concepts and provides explanations for the generated concepts.The method illustrates generated features for multiple text fragments, sometimes with explanations drawn from Wikipedia.
  • Qualitative Analysis of Feature Generation: Generated features usually add relevant domain connections, such as competitors, subsidiaries, technologies, and general concepts for a Wal-Mart headline.The listed features include Sam Walton, Sears Holdings Corporation, ASDA, RFID, Hypermarket, and Chain store, with explanations of their relations to Wal-Mart.
  • Qualitative Analysis of Feature Generation: Bank of America and Bank of Amazon receive distinct concept sets, showing context-sensitive disambiguation of the word “bank.”The financial phrase yields banking concepts, whereas the Amazon phrase yields river- and rainforest-related concepts.
  • Qualitative Analysis of Feature Generation: The method is not fully accurate: word-overlap matching can generate somewhat relevant, irrelevant, or spurious concepts, which feature selection can filter.The paper attributes these errors to bag-of-words matching and retains selected features that are informative for category prediction.
  • The Effect of Feature Generation: Wikipedia features improved categorization for virtually all datasets, with improvements up to 30.4% for RCV1 and 18% for OHSUMED.The Wikipedia-based classifier was significantly superior to the baseline in both micro- and macro-averaged cases, with p < 10^-5.
  • The Effect of Feature Generation: A newer Wikipedia snapshot produced a small but consistent improvement, while Wikipedia results were competitive with and slightly better than those using ODP.The paper also notes that Wikipedia is continuously updated whereas ODP was virtually frozen.
  • Classifying Short Documents: Feature generation produced greater improvement on short documents in most cases, and the classifier significantly outperformed the baseline with p < 2 · 10^-6.The authors relate this result to short documents being inadequately represented by the standard bag of words.
  • Using Inter-article Links as Concept Relations: Inter-article links yielded further improvements on short-document categorization, with more general linked features superior in the absolute majority of cases.The links are treated as relations between concepts represented by Wikipedia articles.

5. Related Work

ESA differs from prior semantic representations by using Wikipedia-derived, human-defined knowledge concepts rather than only statistical, lexical, or manually encoded resources. It also applies uniformly to words and text fragments, supporting richer semantic relatedness and disambiguation.

  • ESA methodology: ESA represents text meaning as weighted vectors of unambiguous human-defined knowledge concepts corresponding to Wikipedia articles.Unlike LSA, its concepts are grounded in manually encoded human knowledge rather than derived from corpus statistics.
  • Text coverage: Unlike methods restricted to individual words or short snippets, ESA treats words and texts in essentially the same way.Other approaches require additional complexity to compare longer texts or are limited by search-engine query length.
  • Empirical comparison: ESA’s richer representation yields much better semantic relatedness results than the compared prior methods.For similarity-only benchmarks, lexical techniques can still have a slight edge over ESA-Wikipedia.
  • Lexical resources: Compared with WordNet-based approaches, ESA uses context to support word sense disambiguation and handles both short and long text fragments.WordNet approaches are constrained by limited information accompanying synsets.

6. Conclusions

The paper presents Explicit Semantic Analysis, which represents text in a high-dimensional space of Wikipedia-based knowledge concepts and enriches conventional text representations. Evaluations report improvements in semantic relatedness and text categorization, while the approach depends on the uneven quality of Wikipedia content.

  • Conclusions: ESA represents text meaning in a high-dimensional space of knowledge concepts corresponding to Wikipedia articles.The representation taps collective human knowledge and includes information not deducible from the input text alone.
  • Interpretation: ESA’s concept-based representation can partially address synonymy and polysemy by enriching texts with related concepts.For example, it generated the concept Extrasolar planet for a passage that did not contain those words.
  • Method: ESA uses standard text classification techniques to match document texts with relevant Wikipedia articles without deep language understanding or specialized inference rules.This provides an automatic way to use encyclopedia knowledge directly.
  • Results: Correlation with human relatedness judgments increased from r = 0.56 to 0.75 (Spearman) for individual words and from r = 0.60 to 0.72 (Pearson) for texts.ESA also provided a uniform method for comparing individual words and arbitrarily long text fragments.
  • Results: Using Wikipedia for feature generation improved text categorization performance across a diverse collection of datasets.The conclusion describes these improvements as consistent despite small gains in recent text categorization research.
  • Caveat: Wikipedia articles are not uniformly high quality, including incomplete, incorrect, non-consensus, or spam-prone content.The authors believe the method is not overly susceptible provided that the majority of content is correct.
  • Extensions: ESA can be adapted to other languages by using the corresponding language’s Wikipedia version and to specialized domains or other knowledge repositories.The paper also reports applications beyond its evaluated tasks, including information retrieval and German word similarity.

Appendix A. The effect of knowledge breadth in text categorization

Using a newer, larger Wikipedia snapshot for feature generation produced slightly greater average text categorization improvements, consistently across datasets.

  • Knowledge breadth: A newer Wikipedia snapshot led to greater average improvements in text categorization performance.The difference between snapshots was small but consistent across datasets.

Appendix B. Test Collections for Text Categorization

The appendix provides detailed descriptions of the test collections used to evaluate knowledge-based feature generation for text categorization.

  • Test collections: The test collections support evaluation of knowledge-based feature generation for text categorization.The appendix contains detailed descriptions of these collections.

B.1 Reuters-21578

Reuters-21578 is a labeled newswire collection used with the ModApte split and selected category sets for text categorization. Its skewed labels, overlapping categories, and informal documents make classification difficult.

  • Experimental split: The ModApte split assigns the first 9,603 documents to training and the remaining 3,299 to testing chronologically.Experiments use the 10 largest categories and 90 categories represented in both training and testing.
  • Category structure: The collection has 118 categories with highly skewed frequencies, ranging from 3,964 positive examples for “earn” to one example in 16 categories.Several category subsets are defined, including the 10 largest, 90 shared training/testing categories, 115 categories with training examples, and all 118 categories.
  • Classification challenges: Overlapping category scopes make classification harder because a document posted to one group may also fit another.Examples include closely related hardware and religion newsgroups.
  • Document characteristics: Informal postings contain abbreviations, foreign words, proper names, and substantial markup characters.These properties distinguish the documents from standardized prose and complicate text processing.

B.3 Movie Reviews

Movie Reviews provides a binary sentiment-classification task rather than standard topical categorization, while prior work used both unsupervised semantic orientation and classical text categorization.

  • Dataset: The Movie Reviews collection contains 1,400 reviews, split evenly between positive and negative movie sentiment.The reviews were collected from the rec.arts.movies.reviews newsgroup and archived at the Internet Movie Database.
  • Task: Its classification objective is to determine whether a review expresses positive or negative sentiment about a movie.This differs from standard topical text categorization.
  • Related methods: Prior semantic-orientation studies estimated document polarity using latent semantic indexing and the reference words “excellent” and “poor.”Another cited study applied classical text-categorization techniques.

B.4 Reuters Corpus Version 1 (RCV1)

RCV1 is a large, hierarchically labeled Reuters corpus, but these experiments use corrected RCV1-v2 data and restricted Topic and Industry subsets for tractable evaluation.

  • Topics: Topic labels comprise 103 hierarchical codes averaging 3.24 categories per document, with broad categories dominating micro-averaged scores.The Hierarchy Policy assigns ancestors when a category is assigned, while the Minimum Code Policy requires at least one Topic and Region code.
  • Industries: Industry labels are finer-grained than Topics and harder to classify, with 351,761 documents carrying Industry codes.Industry categories are also hierarchical, though their hierarchy policy was only partially enforced.
  • Experimental setup: Experiments use 17,808 training and 5,341 testing documents, restricted to 16 Topic and 16 Industry categories.The subset uses adjacent dates to speed experimentation.
  • Data correction: All experiments use RCV1-v2, which corrects distribution errors involving coding and hierarchy-policy violations.RCV1-v1 denotes the original distribution.

B.5 OHSUMED

OHSUMED is a large medical-reference collection with many MeSH labels; the experiments use a 20,000-document subset and randomized ten-category sets.

  • Collection: OHSUMED contains 348,566 medical-journal references published from 1987 to 1991, with 233,445 including abstracts.Each reference is labeled with several MeSH categories.
  • Label structure: The collection has over 14,000 distinct categories and averages 13 categories per document.This creates a highly multi-label categorization setting.
  • Experimental setup: The experiments select 10,000 documents for training and the next 10,000 for testing from 1991 references with abstracts.Five random sets of 10 categories are generated to limit the experimental category space.
  • Evaluation: Micro-averaged scores can be much higher than macro-averaged scores when frequent categories dominate evaluation.The cited discussion specifically notes this effect for RCV1 Topic codes.

Appendix C. Additional Examples of Feature Generation for Text Categorization

The appendix illustrates feature generation across medical, biological, historical, scientific, astronomical, dietary, and ambiguous texts. The examples show that generated Wikipedia concepts reflect textual context and can incorporate related concepts through inter-article links.

  • Feature-generation examples: Generated features connect medical text to leukemia, immune deficiency, cancer, transplantation, and related diseases.The example also identifies a cancer type and an ICD disease code among the selected explanations.
  • Feature-generation examples: Historical and biographical passages generate event-specific concepts such as September 11 memorials, the World Trade Center, Oklahoma City bombing, and related sites.The Iraq example similarly produces concepts including the Iraq disarmament crisis, Yellowcake forgery, Iraq war, and Operation Desert Fox.
  • Feature-generation examples: For ambiguous terms, context selects distinct concept neighborhoods: Jaguar car text yields vehicle models and manufacturers, while Jaguar animal text yields feline taxa and mammals.The car example includes Jaguar models, Daimler Motor Company, British Leyland Motor Corporation, luxury vehicles, and a V8 engine; the animal example includes Felidae and related species.
  • Feature-generation examples: Text about artificial intelligence produces concepts spanning AI subfields, researchers, programming languages, logic, and natural language processing.Link-based generation adds concepts such as Robot, John McCarthy, Artificial consciousness, Marvin Minsky, and Lisp programming language.
  • Feature-generation examples: Astronomical text produces direct topic concepts such as planet, solar system, astronomy, planetary orbit, and extrasolar planet, while linked features add related celestial objects.The linked list includes asteroid, Earth, Oort cloud, comet, Sun, Saturn, Moon, Mercury, asteroid belt, and orbital period.
  • Feature-generation examples: Diet and health text yields concepts covering nutrition, obesity, eating disorders, diseases, food, and health-related conditions.Regular, linked, and more-general features include veganism, diabetes mellitus, healthy eating, body mass index, cancer, food, and McDonald’s.

Appendix D. Comparing Knowledge Sources for Feature Generation: Wikipedia versus WordNet

This appendix compares WordNet- and Wikipedia-based feature generation using a Wal-Mart sentence. It reports that Wikipedia overcomes WordNet’s context-disambiguation and proper-name limitations.

  • Comparison: The comparison juxtaposes WordNet-based and Wikipedia-based feature generation for the same sample sentence.The sentence is “Wal-Mart supply chain goes real time.”
  • Wikipedia: Wikipedia features for the Wal-Mart sentence include the founder, major competitors, and RFID, a technology Wal-Mart uses extensively to manage stock.The listed concepts are Sam Walton; Sears Holdings Corporation, Target Corporation, and Albertsons; and RFID.
  • WordNet: WordNet represents terms through synsets and immediate hypernyms, producing broad and diverse classifications when context words cannot be properly disambiguated.The appendix states that WordNet version 1.6 was used and illustrates noun and verb synsets with more-general synsets.
  • Comparison: The appendix attributes Wikipedia’s advantage to overcoming WordNet’s disambiguation shortcomings and its lack of proper names.It states that Wikipedia encodes a greater wealth of information and that the proposed methodology does not suffer from these shortcomings.
Loading 1401.5697v1…