Source-linked AI summary
Mining Meaning from Wikipedia
Olena Medelyan, David Milne, Catherine Legg, Ian H. Witten
TL;DR
Research has increasingly used Wikipedia’s large, structured accumulation of concepts and relations, but questions remain about its reliability and coverage. This paper comprehensively surveys how Wikipedia is mined and adapted across language, retrieval, extraction, and ontology-building applications. It concludes that Wikipedia has immense potential as a repository of linguistic knowledge, with remarkably high performance reported for tasks such as word sense disambiguation and word similarity, while noting that larger evaluation and training datasets are still needed.
Problem
Research needed a comprehensive account of how Wikipedia’s concepts, relations, facts, and descriptions could be extracted and used across multiple applications.
Method
The paper surveys research using, improving, adapting, and combining Wikipedia across natural language processing, information retrieval, information extraction, and ontology building.
Results
Wikipedia shows immense potential as a repository of linguistic knowledge, with automatically addressing word sense disambiguation and word similarity at a remarkably high performance level.
Takeaways & Limitations
Wikipedia is a promising resource for linguistic research and applications that exploit its concepts, relations, facts, descriptions, and structure.
Takeaways & Limitations
Larger datasets are needed for evaluation and training to advance research further.
Abstract
from arXiv · showhide
Wikipedia is a goldmine of information; not just for its many readers, but also for the growing community of researchers who recognize it as a resource of exceptional scale and utility. It represents a vast investment of manual effort and judgment: a huge, constantly evolving tapestry of concepts and relations that is being applied to a host of tasks. This article provides a comprehensive description of this work. It focuses on research that extracts and makes use of the concepts, relations, facts and descriptions found in Wikipedia, and organizes the work into four broad categories: applying Wikipedia to natural language processing; using it to facilitate information retrieval and information extraction; and as a resource for ontology building. The article addresses how Wikipedia is being used as is, how it is being improved and adapted, and how it is being combined with other structures to create entirely new resources. We identify the research groups and individuals involved, and how their work has developed in the last few years. We provide a comprehensive list of the open-source software they have produced.
1. INTRODUCTION
This paper surveys research that mines Wikipedia’s concepts, relations, facts, and descriptions for language and information applications. It presents Wikipedia as a middle ground combining large scale with explicit structure, while noting important quality and coverage limitations.
- 1. INTRODUCTION: The paper provides the first comprehensive summary, up to mid-2008, of research mining meaning from Wikipedia.Its scope includes concepts, topics, descriptions, facts, and relations.
- 1. INTRODUCTION: Traditional hand-crafted resources are generally high quality but restricted in size and coverage because they rely on expert input.Experts cannot keep abreast of the constant growth of new discoveries and topics.
- 1. INTRODUCTION: Statistical approaches to unstructured text can emulate human intelligence for particular purposes, but cracks appear when generalizing across domains and tasks.The passage attributes this boundary to inconsistency in human language and limits of replicating intuitive understanding with rules or trends.
- 1. INTRODUCTION: Wikipedia offers a middle ground between hand-crafted resources and large-scale analysis of unstructured text.It combines the scale of broad data with manually created structure.
- 1. INTRODUCTION: Wikipedia contains around 18 Gb of text and structural features such as links, categories, and infoboxes that provide explicitly defined semantics.These features distinguish it from corpora lacking comparable explicit structure.
- 1. INTRODUCTION: The survey covers Wikipedia’s use in natural language processing, information retrieval, information extraction, and ontology building.It also describes the people, research groups, and open-source resources involved.
2 WIKIPEDIA: A RESOURCE FOR MINING MEANING
Wikipedia combines exceptional scale with open, collaborative editing, making its concepts, relations, and structure valuable for mining meaning. Researchers exploit these resources while addressing their changing content, evaluation challenges, and biases.
- Resource scale: Wikipedia’s exceptional scale and multilingual coverage make it a substantial resource for extracting concepts, relations, and facts.Its English articles alone are ten times the size of Encyclopedia Britannica, while the project spans hundreds of languages.
- Editing and trust: Open collaborative editing initially raised concerns about vandalism, misinformation, and destructive edit wars.These pressures led to increasingly sophisticated policies, Wikiquette, administrators, and mechanisms for managing disruptive editing.
- Mining challenges: Wikipedia’s continuously changing content can make mining results stale and complicates evaluation when independent ground truth is unavailable.The article identifies version change and topics documented only by Wikipedia editors as central methodological challenges.
- Structural signals: Articles, hyperlinks, redirects, categories, and other structure support semantic mining without always requiring external resources.Hyperlinks provide synonyms beyond redirects, while article structure can be mined for meaning and used to link Wikipedia with other datasets.
- Quality and bias: Mining and quality studies report both strong performance and important variation across measures, domains, and perspectives.One comparison found 162 mistakes in Wikipedia versus 123 for Britannica, while coverage was described as representative except in law and medicine; PageRank and HITS also surfaced different kinds of information.
- Research uses: Wikipedia has supported applications in language processing, retrieval, information extraction, and ontology construction.Its Wiki technology is described as a large-scale collaborative ontology development environment, with techniques that generate information for articles lacking it.
3 SOLVING NATURAL LANGUAGE PROCESSING TASKS
Wikipedia has been applied across natural language processing tasks as an alternative or complement to manually constructed resources. Across semantic relatedness, disambiguation, linking, and co-reference, reported results show strong performance alongside task- and dataset-specific limitations.
- Semantic relatedness: Semantic relatedness results are dataset-dependent: WordNet-based measures lead on M&C and R&G, while Wikipedia wins by a large margin on WS-353.Combining Wikipedia and WordNet evidence with an SVM achieved a correlation of 0.62 on the WS-353 testing subset.
- Overview: Wikipedia can serve AI applications similarly to hand-crafted knowledge resources and often significantly outperforms WordNet on shared tasks.Direct comparisons support comparable use, with stronger performance reported in many cases.
- Semantic relatedness: ESA is the strongest Wikipedia-based relatedness method described, reaching 0.75 average correlation on WS-353 and supporting words, phrases, and documents.The corresponding Open Directory Project result is 0.65 correlation, while Wikipedia-based measures cover a larger vocabulary than resources such as WordNet.
- Word sense disambiguation: Document-specific adjustment of similarity and prior probability raises linking F-measure from 93% to 97%.The approach adjusts the relative importance of these signals from document to document.
- Word sense disambiguation: Wikipedia-based word-sense disambiguation and entity-linking systems report accuracies from 55% to 85%, with individual studies reaching 88%, 91%, 92%, and 93%.Reported results vary with the model, data, task, and mapping setting.
- Co-reference resolution: Wikipedia-derived features improve co-reference results overall, although they decrease precision on NWIRE while increasing recall on both datasets.Reported overall gains are 1.5 to 2 points on NWIRE and 6 points on BNEWS.
4. INFORMATION RETRIEVAL
Wikipedia is used to understand queries and documents, expand searches, support multilingual retrieval, answer questions, and improve access to information. Across these applications, its concepts, links, redirects, categories, and multilingual structure provide useful retrieval signals, though effectiveness varies by task and evaluation setting.
- Query expansion: Wikipedia-derived topics and links expand queries with synonyms, alternative spellings, related concepts, and Boolean reformulations.Koru maps query phrases to Wikipedia topics, mines redirects, and adds related terms and quoted phrases.
- Query expansion: 95% of queries were recognized, and expanded queries significantly outperformed original queries in overall F-measure.The result was reported for evaluation on the TREC HARD Track.
- Query expansion: Wikipedia-based expansion improved queries where pseudo-relevance feedback performed poorly, but did not achieve state-of-the-art performance overall.The authors attribute this limitation to language and context differences between Wikipedia and dated news articles.
- Concept-based retrieval: Concept-based retrieval represents documents and queries through relevant Wikipedia articles and combines those scores with traditional retrieval scores.ESA-based representations resolve variable terminology by comparing concepts as well as words.
- Concept-based retrieval: 4–15% improved Mean Average Precision was obtained when Wikipedia-based concept scores augmented different retrieval systems.Wikipedia’s breadth supports term-concept mapping through articles, redirects, and disambiguation pages.
- Question answering: Question-answering systems mine Wikipedia’s facts and structure to retrieve or construct answers, with one system improving 4.5% over its original version.Other evaluations found that supervised use was important and that answer quality varied across resources and languages.
World Wide Web.
Wikipedia supports entity ranking, topic organization, and text categorization by supplying structured links, categories, concepts, and background knowledge. The surveyed work reports gains over established methods, while noting sparse evaluation and the need for larger datasets.
- Entity ranking: Wikipedia-based entity ranking combines search-engine scores with category and inter-article-link features, nearly doubling precision over search alone.The link features provide a simplified PageRank-like signal, while categories supply similarity information.
- Entity ranking: Wikipedia-derived entity-ranking features have not yet been evaluated against more sophisticated baselines or query-specific significance measures.Research on entity ranking over Wikipedia remains sparse and young.
- Topic organization: Wikipedia articles and WikiBooks can organize entities into hierarchical topic maps and generate domain-specific taxonomies, but these methods were not evaluated.The approach searches for relevant articles and books, then organizes linked entities under their source headings.
- Open limitations: Larger datasets are needed for future evaluation and training, and several Wikipedia-based organization tasks remain insufficiently investigated.This scope boundary applies particularly to topic indexing and entity organization.
- Text categorization: Wikipedia-based categorization has improved upon a performance plateau reached by established techniques, with gains especially reported for short documents and smaller categories.The surveyed research area is described as flourishing, with many recent efforts improving previous state of the art.
- Text categorization: Wikipedia-based text categorization augments documents with relevant concepts, helping address terminology differences and limited features in short documents.Concept features can be derived without complex natural language processing and combined with ordinary word representations.
- Text categorization: A news-story clustering system achieved 90% accuracy using only titles and descriptions, while Wikipedia-based approaches produced the best overall performance in Table 4.Wikipedia supplies background knowledge about the concepts associated with words and phrases in category labels.
5. INFORMATION EXTRACTION AND ONTOLOGY BUILDING
This section surveys methods that extract relations and entity types from Wikipedia text and structure, then organize them into thesauri, taxonomies, and ontologies. Results range from improved retrieval and relation extraction to large machine-readable knowledge resources, although evaluation and relation definitions remain inconsistent.
- 5.1 Deriving relations from article text: Information extraction converts textual statements into normalized relations such as hasHeadquarters(Apple Inc., 1 Infinite Loop-Cupertino-California).The central challenge is recognizing equivalent relations across different wording and companies.
- 5.1 Deriving relations from article text: Wikipedia-specific properties, including encyclopedic style, hyperlinks, categories, and infoboxes, support relation extraction and ontology construction.Semi-structured content and internal links can provide more semantic relations than article text.
- 5.1 Deriving relations from article text: 1200 new relations were identified with 61–69% precision depending on relation type, while restricting extraction to relevant categories increased player-team precision to 93%.Automatically extracted patterns raised recall from 14% at 92% precision to 37% at 65% precision.
- 5.3 Ontology building: The surveyed resources differ in extracted relation inventories, and the lack of a common evaluation basis prevents conclusions about which approach performs best.Researchers also lack consensus on which relations to extract, with overlaps among relations such as isMemberOf, instanceOf, and hasDirector.
6. PEOPLE, PLACES AND RESOURCES
Research on mining Wikipedia is distributed across institutions worldwide, spanning language processing, retrieval, relation extraction, and ontology construction. The section also catalogs open-source tools, demonstrations, and datasets that operationalize this work.
- Research groups: Researchers across North America, Europe, Asia, Australia, and New Zealand apply Wikipedia to entity recognition, semantic relatedness, retrieval, relation extraction, and knowledge bases.The surveyed community includes institutions in the US, Germany, Spain, the Netherlands, France, the UK, Israel, India, China, Japan, New Zealand, and Australia.
- Research groups: The University of Waikato covers entity recognition, query expansion, topic indexing, semantic relatedness, and augmentation of existing knowledge bases.RMIT collaborated with INRIA on entity ranking.
- Tools and resources: Table 8 organizes resources into tools for accessing and processing Wikipedia, mining demonstrations, and datasets generated from Wikipedia.The listed resources include APIs, analytic tools, query systems, thesauri, dictionaries, and annotated corpora.
- Tools and resources: JWPL provides structural access to redirects, categories, articles, and link structure through a Java API.Other listed systems provide semantic relatedness, preprocessing, query answering, query expansion, and text annotation.
- Tools and resources: The catalog includes DBpedia access to 103M extracted facts, a YAGO demo, and a large-scale thesaurus containing 78M associations.Additional resources include bilingual dictionaries, location data, named-entity annotations, taxonomies, and mappings to Cyc.
7. SUMMARY
The survey presents Wikipedia as a large, evolving source for linguistic and encyclopedic knowledge that supports NLP, retrieval, extraction, and ontology building. Its strongest evidence concerns language applications and document classification, while several tasks and evaluation standards remain underdeveloped.
- Overall assessment: Wikipedia unexpectedly functions as a substantial repository of linguistic knowledge despite being created primarily for encyclopedic coverage.Its descriptions, links, categories, redirects, disambiguation pages, and other structures provide multiple mineable information sources.
- Natural language processing: Wikipedia enables automatic word-sense disambiguation and word-similarity methods with remarkably high performance.Extended concept descriptions, hyperlinks, and implicit conditional probabilities of word meanings support these applications.
- Open directions: Research has mostly focused on English, while natural-language generation, machine translation, and discourse analysis have not yet been explored as Wikipedia applications.The survey describes these areas as open opportunities for future work.
- Information retrieval: Document classification and topic indexing have produced significant advances over the state of the art by exploiting Wikipedia’s distinctive properties.Question answering and entity ranking are less well addressed because current techniques often treat Wikipedia as just another corpus.
- Information extraction and ontology building: Information-extraction and ontology-building approaches produce resources ranging from hundreds to millions of relations, but their tasks and extracted relation types vary substantially.The absence of a common evaluation basis prevents an overall judgment about which approach performs best.
- Future of Wikipedia: The survey highlights a tension between making Wikipedia more structured and preserving its public, amateur-editor model.It frames the interaction between formal semantics and community editing as an open question.