PaperlayerView exact revision on arXiv

Source-linked AI summary

Analysis of Named Entity Recognition and Linking for Tweets

Leon Derczynski, Diana Maynard, Giuseppe Rizzo, Marieke van Erp, Genevieve Gorrell, Raphaël Troncy, Johann Petrak, Kalina Bontcheva

arXiv:1410.7182v1cs.CL

TL;DR

Twitter's short, noisy, context-dependent text makes reliable named entity recognition and linking difficult, while existing systems were largely developed for longer news content. The paper introduces a new Twitter entity-disambiguation dataset and empirically evaluates NER and NEL systems, finding that microblog performance remains less robust than newswire performance and is especially affected by poor capitalisation. It also identifies annotation and coverage boundaries that constrain evaluation.

  • Problem

    Reliable NER and NEL on short, noisy tweets remain limited because systems are commonly developed for longer news text and tweets provide less disambiguating context.

  • Method

    The paper constructs a Twitter NEL dataset and evaluates general-purpose, domain-specific, word-level, and whole-tweet NER/NEL approaches.

  • Results

    Twitter-specific methods can exceed 80% F1, but NER and NEL remain below newswire state-of-the-art, with poor capitalisation causing particularly significant NER recall drops.

  • Takeaways & Limitations

    Poor capitalisation is the most significant identified factor affecting NER recall, while typographic errors, out-of-vocabulary words, shortenings, and slang also impair performance.

  • Takeaways & Limitations

    Whole-tweet linking measurements are only rough indicators because topic annotations may not identify entities and often omit other entities in the tweet.

Abstract

from arXiv · show

Applying natural language processing for mining and intelligent information access to tweets (a form of microblog) is a challenging, emerging research area. Unlike carefully authored news text and other longer content, tweets pose a number of new challenges, due to their short, noisy, context-dependent, and dynamic nature. Information extraction from tweets is typically performed in a pipeline, comprising consecutive stages of language identification, tokenisation, part-of-speech tagging, named entity recognition and entity disambiguation (e.g. with respect to DBpedia). In this work, we describe a new Twitter entity disambiguation dataset, and conduct an empirical analysis of named entity recognition and disambiguation, investigating how robust a number of state-of-the-art systems are on such noisy texts, what the main sources of error are, and which problems should be further investigated to improve the state of the art.

1. Introduction

Information extraction from Twitter is difficult because microblogs are short, noisy, ambiguous, and unlike the longer news text used to develop many systems. The paper addresses this challenge by introducing a Twitter entity-linking dataset and evaluating, analysing, and comparing NER and NEL approaches.

  • Information extraction produces fixed-type, unambiguous snippets from text for display, storage, search, and other information-access tasks.
  • NER identifies and types entities such as people, locations, organisations, and products, while NEL links entity mentions to entities or external resources.
  • Microblogs are challenging because their brevity limits context and makes ambiguity difficult to resolve, while their noisy language differs from carefully written news text.
  • NER methods typically achieve 85-90% accuracy on longer texts but only 30-50% on tweets.
  • The paper constructs a Twitter NEL dataset, evaluates general-purpose and domain-specific NER and NEL systems, and analyses errors, normalisation, and pipeline configurations.

2. Named Entity Recognition

The paper evaluates general-purpose, Twitter-specific, commercial, and hybrid NER systems across three tweet datasets, combining performance comparisons with error analysis. Results show that microblog adaptation helps, but NER remains substantially weaker than on longer text and is affected by noisy, transient, and context-poor content.

  • Systems evaluated: The evaluation compares Twitter-specific, general-purpose, commercial, and hybrid NER tools using different recognition approaches and Web APIs.The systems include gazetteer- and finite-state methods, machine-learning models, topic-model pipelines, and the Twitter-tailored NERD-ML approach.
  • Datasets and evaluation: The tools were evaluated on three available Twitter datasets whose differing entity schemes were mapped to person, location, organisation, and miscellaneous.The datasets include the Ritter corpus, UMBC crowdsourced gold data, and MSM2013; the mappings complicate direct comparative evaluation.
  • Results: Conventional newswire-trained tools perform poorly on tweets, making microblog domain adaptation crucial, while overall performance remains below that typically achieved on longer texts.The results include substantial missed entity mentions and false positives.
  • Limitations: Interpretation is constrained by imperfect entity-label mappings, single-annotator data, and evaluation of tweet text without users’ external or temporal context.These factors can introduce bias and make statistical differences difficult to discern while making recognition harder than for the intended reader.
  • Results: On UMBC, the NERD-ML chain achieved the best recall, but its 31.02% recall illustrates the dataset’s difficulty.The corpus contains many transient entities, including pop music bands, and may have deliberately favored difficult cases.
  • Results: NERD-ML achieved the best performance on all three scored MSM2013 entity types, while Stanford NER achieved the best precision.The MSM2013 evaluation excluded MISC entities because no deterministic mapping to the other classes was possible.

3. Entity Linking

Entity linking in microblogs is underexplored and difficult because tweets provide limited contextual evidence for disambiguation. The paper distinguishes whole-tweet “aboutness” linking from explicit-mention linking and evaluates several systems against the latter task.

  • Microblog NEL remains relatively new and underexplored, with insufficient context posing a central disambiguation problem.
  • Whole-tweet NEL, or “aboutness,” identifies the topics and entities that best capture a microtext’s meaning, including implicitly relevant entities.For example, a tweet mentioning iPad may make Apple relevant without explicitly naming it.
  • Explicit-mention NEL requires identifying entity boundaries and assigning each mention a unique target identifier.
  • YODIE links mentions to DBpedia URIs using string, semantic, contextual, and Wikipedia-frequency metrics.Its evaluation combines a Twitter tokeniser and POS tagger with the ANNIE NER system.
  • The evaluation compares YODIE with generic systems including DBpedia Spotlight, Zemanta, and TextRazor.

3.1. Word-level Entity Linking

Word-level entity linking in tweets is difficult because it depends on accurate recognition, sufficient context, and suitable disambiguation resources. The paper constructs a public DBpedia-linked corpus, evaluates systems, and identifies contextual, capitalization, and coverage-related limitations.

  • Word-level linking is difficult because unrecognised expressions cannot be disambiguated, while short tweets often lack enough context even for humans.
  • A shared manually disambiguated corpus was lacking, limiting reliable evaluation of word-level entity linking on microblogs.
  • The authors created a public corpus of 182 microblog texts containing entity mentions assigned DBpedia URIs.It was built from 177 selected mentions and volunteer expert annotations.
  • Short collection periods may limit generalisation because entity prevalence in social media changes over time.The corpus was used for evaluation rather than training, but it does not represent a drift-free evaluation scenario.
  • YODIE performed best among the compared systems, partly attributed to its Twitter-adapted preprocessing components.
  • Unreliable capitalization, unconventional entity uses, and entities absent from DBpedia create major recognition and linking errors.False positives were concentrated in sentence-case words, whereas false negatives were concentrated in lowercase words.

3.2. “Whole tweet” Entity Linking

Whole-tweet entity linking is evaluated on a topic-disambiguated dataset, but its annotations do not consistently represent actual entity mentions. Results therefore provide only a rough and somewhat inconclusive comparison of systems.

  • The topic-disambiguated dataset links tweets to corresponding Wikipedia articles, but topic labels may describe the text rather than an entity.
  • Only top-ranked entities were matched when evaluating DBpedia Spotlight, Zemanta, and TextRazor, and recognised but unlinked entities were discarded.
  • Zemanta achieved the highest F1 score, while DBpedia Spotlight had the highest recall and TextRazor had the highest precision.
  • The comparison is somewhat inconclusive because tweet-level annotations may omit entities present in the text and vary in specificity.

3.3. Discussion

The discussion identifies limited context, implicit semantics, noisy capitalization, and insufficient evaluation data as central obstacles to tweet entity linking. The authors respond by creating and expanding an expert-annotated DBpedia gold standard.

  • Tweet entity linking is challenging because tweets provide limited surrounding context and encode semantics implicitly through hashtags and user mentions.
  • Reliable comparison of methods is difficult because few suitable datasets are available for this genre.
  • The authors created a small expert-annotated DBpedia gold standard and planned expansion through paid CrowdFlower annotation.
  • Inconsistent tweet capitalization causes performance losses because capitalization is less reliable as a proper-noun indicator than in news text.
  • The discussion attributes current difficulties partly to prior NER problems, limited gold-standard data, and tweets’ short, noisy, context-dependent nature.

4. Analysis

Microblog characteristics make NER and NEL substantially harder than processing longer, well-formed text. The analysis identifies capitalization, typographic noise, compressed language, limited context, multilingualism, and scarce annotated resources as central challenges.

  • 4. Analysis: Microblog-specific characteristics create difficulties across NER and NEL, with errors arising from the genre’s shortness, noise, compressed language, and limited context.The analysis also identifies multilingual and user-generated content as additional challenges.
  • Multilingual: Less than 50% of tweets are in English, while information extraction methods have mostly focused on English and low-overhead adaptation remains unresolved.Language identification is therefore an important first step before applying language-specific processing.
  • Capitalisation: Capitalization errors affect POS tagging, NER, and entity linking by removing distinctions between common and proper nouns.Named entities in social media are more likely to appear in lowercase than in newswire text.
  • Capitalisation: 89.2 precision and 88.5 recall on uncased newswire fell to 81.7 precision, 4.1 recall, and 7.8 F1 after lowercasing.The result demonstrates the particularly severe effect of capitalization information on recognition.
  • Noise and compressed language: Typographic errors confuse tokenization, POS tagging, and NER, while current normalization poorly balances repairing mistakes against preserving correct OOV words.Microblog shortness also encourages rare, incomplete, and abbreviated grammatical structures that are difficult to interpret without context.
  • Typographic noise: Social-media text has an OOV rate 2-2.5 times that of newswire, although the MSM corpus achieved the best NER results despite its highest OOV rate.This indicates that vocabulary coverage alone does not explain corpus-level NER performance.
  • Context and resources: Entity linking is especially difficult because microblogs often lack enough surrounding context to resolve polysemous mentions, even for human annotators.Larger annotated linguistic resources are needed because annotated microblog text is rare.

5. Reducing Microblog Noise through Pre-Processing

The paper evaluates preprocessing adaptations intended to reduce microblog noise before NER and NEL. Language identification improves with Twitter adaptation, while POS adaptation and normalization provide only limited or conditional gains.

  • Overview: Preprocessing errors caused by microblog noise cascade through language detection, tokenization, POS tagging, NER, and NEL.The study therefore compares genre-adapted preprocessing methods with generic alternatives.
  • 5.1. Language Identification: Microblog-adapted TextCat performed consistently better than the other language-identification systems.Adapted langid.py selected a language outside the five evaluation choices for 6.3% of tweets and performed worse than its generic version.
  • 5.1. Language Identification: Language identification performance was sufficiently high to inform reliable choices in later processing stages, despite being harder on tweets than longer texts.The comparison is affected by TextCat choosing among five languages while adapted langid.py chooses among 97.
  • 5.2. POS Tagging: Twitter-specific POS tagging produced a small NER benefit, but recall remained low.ANNIE’s gazetteer-based design is less sensitive to some POS errors, while unseen items limit its recall.
  • 5.2. POS Tagging: Microblog-adapted POS tagging reduced errors but still performed below longer, cleaner content, negatively affecting NER and NEL.The remaining gap indicates that domain adaptation alone does not resolve downstream recognition problems.
  • 5.3. Normalisation: Full normalization slightly reduced false positives, whereas basic normalization lowered overall performance by causing more entities of other types to be missed.The Stanford NER system benefited only from strong normalization, reflecting its reliance on context.
  • 5.3. Normalisation: Normalization helps but is insufficient as the sole method for combating microblog noise.The paper evaluates both heuristic and gazetteer-based approaches for improving entity extraction and linking.

6. Conclusion

The paper concludes that current NER and NEL systems are not robust on terse, noisy microblogs, although Twitter-specific methods and preprocessing yield some improvements. Progress requires microblog-specific algorithms, more context, and substantially larger annotated resources.

  • 6. Conclusion: State-of-the-art NER and NEL approaches do not perform robustly on ill-formed, terse, and linguistically compressed microblog text.Some Twitter-specific methods exceed 80% F1 but remain below newswire performance.
  • 6. Conclusion: Poor capitalization causes the largest reported performance drops, while typographic errors, OOV words, shortenings, and slang have less pronounced effects.The capitalization effect also appears on lowercased newspaper content.
  • 6. Conclusion: Language identification, microblog-trained POS tagging, and normalization produce some improvements, but precision and recall remain low.The authors report ongoing work on a microblog-specific recaser to address capitalization.
  • 6. Conclusion: Microblog-specific NER and NEL algorithms are needed to reach the precision and recall achieved on news articles.The conclusion attributes the continuing difficulty partly to the different kinds of entities typical in microblogs.
  • 6. Conclusion: Entity linking remains particularly challenging because surrounding microblog context is often insufficient even for human annotators.Structured knowledge bases such as Wikipedia are proposed as one way to counterbalance the lack of context.
  • 6. Conclusion: Fewer than 10,000 tweets are annotated with named entity types, limiting adaptation and parameter tuning for this genre.The paper suggests crowdsourcing, including selective human annotation of difficult cases, as a possible response.
  • 6. Conclusion: Existing corpora hinder comparison through incompatible annotation types and insufficiently standardized tweet-level entity-linking coverage.Longer-genre corpora may aid initial training, but microblog-specific data remains critical.

Vitae

The paper’s authors are researchers working across information extraction, named entity recognition and linking, social media, semantic web technologies, and related NLP applications.

  • Authors: Leon Derczynski researches data-intensive computational linguistics, information extraction, spatiotemporal semantics, and noisy social-media language.He is a post-doctoral Research Associate at the University of Sheffield.
  • Authors: Diana Maynard works on information extraction, opinion mining, social media, and Semantic Web technology, and has led development of Sheffield’s multilingual IE tools.She is a Research Fellow at the University of Sheffield.
  • Authors: Giuseppe Rizzo’s research interests include information extraction, social media, and the Semantic Web.He is affiliated with the University of Torino and EURECOM.
  • Authors: Marieke van Erp researches reconstruction of news stories from daily streams by extracting and merging who, what, where, and when information.She is a postdoctoral researcher on the NewsReader project at VU University.
  • Authors: Genevieve Gorrell works on named entity recognition and linking, machine learning for disambiguation, and biomedical text mining.She is part of the Natural Language Processing research group at the University of Sheffield.
  • Authors: Raphaël Troncy researches Semantic Web and multimedia technologies, knowledge representation, ontology modeling and alignment, and Web Science.He is a primary investigator on national and European projects.
  • Authors: Johann Petrak researches named entity disambiguation and linking, ontology-based information extraction, and semantic search.He is a research fellow at the University of Sheffield.
  • Authors: Kalina Bontcheva works on information extraction, opinion mining, natural language generation, social-media summarization, and NLP infrastructures.She is a senior research scientist and a leading developer of GATE.
Loading 1410.7182v1…