Source-linked AI summary
Ultra-Fine Entity Typing
Eunsol Choi, Omer Levy, Yejin Choi, Luke Zettlemoyer
TL;DR
Existing entity typing models use relatively coarse, often ontology-bound types and do not adequately cover context-sensitive typing for all entity mentions. The paper introduces open-vocabulary typing with crowdsourced ultra-fine labels and trains a multitask model using head-word and entity-linking supervision. The approach improves performance on the new data and an existing fine-grained benchmark, while the evaluation exposes challenges from incomplete labels and fine-grained prediction.
Problem
Existing models predict relatively coarse types and mainly consider named entities, while hand-crafted ontologies limit coverage of fine-grained, context-sensitive types.
Method
The paper predicts free-form noun-phrase types for all entity mentions and combines head-word supervision with traditional entity-linking signals in a multitask objective.
Results
The model improves performance on the new dataset and achieves state-of-the-art performance on an existing fine-grained NER benchmark.
Takeaways & Limitations
Ultra-fine, open-vocabulary typing supports more diverse type coverage and establishes evaluation baselines for context-sensitive entity typing.
Takeaways & Limitations
The model often abstains from fine-grained types or produces incomplete related-type sets, contributing to low recall.
Abstract
from arXiv · showhide
We introduce a new entity typing task: given a sentence with an entity mention, the goal is to predict a set of free-form phrases (e.g. skyscraper, songwriter, or criminal) that describe appropriate types for the target entity. This formulation allows us to use a new type of distant supervision at large scale: head words, which indicate the type of the noun phrases they appear in. We show that these ultra-fine types can be crowd-sourced, and introduce new evaluation sets that are much more diverse and fine-grained than existing benchmarks. We present a model that can predict open types, and is trained using a multitask objective that pools our new head-word supervision with prior supervision from entity linking. Experimental results demonstrate that our model is effective in predicting entity types at varying granularity; it achieves state of the art performance on an existing fine-grained entity typing benchmark, and sets baselines for our newly-introduced datasets. Our data and model can be downloaded from: http://nlp.cs.washington.edu/entity_type
1 Introduction
Ultra-fine entity typing targets context-sensitive, free-form types for all noun-phrase mentions, addressing the coarse, ontology-bound coverage of prior models. The paper combines head-word supervision with entity-linking signals and reports improved performance, including state-of-the-art results on an existing benchmark.
- Motivation: Prior entity typing models predict relatively coarse types and mainly consider named entities, limiting fine-grained inference for mentions such as pronouns and nominals.The paper motivates finer typing for context-sensitive tasks including coreference resolution and question answering.
- Task: The new task predicts free-form noun phrases describing an entity mention’s role in context, covering named entities, nominals, and pronouns.Its large type vocabulary allows the formulation to subsume existing fine-grained named entity typing tasks.
- Motivation: Hand-crafted ontologies provide limited concept coverage, with existing datasets skewed toward coarse labels and OntoNotes assigning about half of mentions to “other.”The paper presents this skew as evidence that predefined ontologies struggle to cover fine-grained concepts.
- Data: The crowdsourced evaluation data contains over 2,500 unique types and is substantially more diverse and fine-grained than previous datasets.Annotators reached 85% agreement when judging whether another annotator’s generated type was appropriate.
- Method: Head words provide contextual distant supervision, complementing context-oblivious entity-linking supervision for open-type prediction.For example, a noun phrase’s head word can directly indicate a type, while context can favor “director” over “mayor” for Clint Eastwood.
- Results: Mixing fine- and coarse-grained supervision in a multitask objective improves performance, and head-word supervision yields state-of-the-art results on an existing fine-grained NER benchmark.The model also establishes baselines for the newly introduced evaluation datasets.
2 Task and Data
The paper defines ultra-fine typing as context-sensitive prediction of natural-language type phrases and builds a crowdsourced, multi-domain benchmark. Its labels and mentions are substantially broader than prior datasets, while the model can predict types unseen in the crowdsourced examples.
- Task: The task predicts a set of natural-language phrases T describing an entity mention e in its sentence context.Context can distinguish an appropriate type such as “philanthropist” from another plausible type such as “inventor.”
- Data: The dataset contains about 6,000 crowdsourced mentions sampled from Gigaword, OntoNotes, and web articles.Mentions were selected from maximal noun phrases and a coreference-resolution system.
- Annotation: The collection process prioritizes precision, so the resulting type set is diverse but not comprehensive.This makes evaluation non-trivial because valid types may be absent from the collected annotations.
- Annotation: Each example averages 5 labels: 0.9 general, 0.6 fine-grained, and 3.9 ultra-fine types.The collection uses five crowd workers and encourages one general plus two specific types from a vocabulary of about 10K frequent noun phrases.
- Open-Type Prediction: The model correctly predicts “television network” and “archipelago” even though those types never appear in the 6,000 crowdsourced examples.This illustrates open-type prediction beyond the observed crowdsourced labels.
- Type Coverage: To cover 80% of examples, the dataset requires 429 different types, compared with 7 in FIGER and 4 in OntoNotes.The result indicates a much less concentrated label distribution than in prior benchmarks.
- Mention Coverage: The benchmark contains roughly 40% pronouns, 38% nominal expressions, and 22% named entity mentions.This broad mention composition supports evaluation beyond predominantly named-entity settings.
3 Distant Supervision
The paper combines entity-linking supervision with contextual signals from head words to expand coverage and provide ultra-fine type labels. Wikipedia definitions improve entity and type coverage, while head-word supervision offers a noisy but effective context-sensitive signal.
- Entity Linking: Entity linking suffers from knowledge-base incompleteness and context mismatch, limiting recall and precision.The paper addresses recall with Wikipedia-linked data and extracts types from encyclopedic definitions.
- Entity Linking: Wikipedia definition mining extracted descriptions for 3.1M entities containing 4,600 unique type labels.The extracted labels include types such as “competition,” “movement,” and “village.”
- Contextualized Supervision: Head words provide an easy-to-gather, context-sensitive type signal for nominal mentions, though they can be noisy.For example, “graphics card” identifies a type for a mention of Titan V.
- Contextualized Supervision: The system extracts nominal head words from Gigaword and Wikilink data, adds compatible neighboring nouns, and normalizes words and plurals.Multiword expressions are retained when they form phrases in the type vocabulary.
- Contextualized Supervision: Head-word supervision has around 80% accuracy, comparable to types extracted through entity linking.Errors arise from parsing, idioms, and transparent heads; multitask training and regularization help the signal encode context.
4 Model
The model predicts type sets from contextualized sentence and mention representations, using label embeddings and a multitask objective across type granularities. Its supervision design avoids treating unobserved types at other abstraction levels as negative evidence.
- Architecture: The model predicts sets of types from mentions in context using an architecture inspired by AttentiveNER and a new multitask objective.The objective handles multiple supervision sources.
- Context Representation: Each token combines a pretrained embedding with a location embedding before bidirectional LSTM contextualization and attention-based pooling.MLP attention forms the sentence context representation from contextualized token states.
- Mention Representation: The mention representation combines character-CNN features with an attention-weighted sum of pretrained word embeddings.The final representation concatenates the context and mention representations.
- Label Prediction: The model predicts each type with a sigmoid over its label embedding’s inner product with the mention representation.It selects every type above 0.5, or the maximum-probability type when none exceeds that threshold.
- Multitask Objective: The multitask objective updates only granularity bins containing at least one positive label, reflecting partial supervision from different sources.A missing type at one abstraction level is not treated as negative evidence for another level.
5 Evaluation
The evaluation combines crowdsourced data with entity-linking, Wikipedia-definition, and head-word supervision to assess ultra-fine entity typing. The model predicts diverse types, but fine-grained recall remains limited by incomplete label prediction and incomplete gold annotations.
- Experiment Setup: The evaluation combines about 2,000 crowdsourced examples with 4.7M entity-linking examples and 20M head-word examples.The crowdsourced dataset is split into train, development, and test sets of about 2,000 examples each.
- Model Results: The model uses a multitask objective to learn finer types without punishing more general types, improving recall while reducing precision.Its MRR is slightly better than the baseline at ranking correct types above incorrect ones.
- Granularity and Supervision: Finer labels are harder to predict than coarse labels, with ultra-fine types posing the greatest challenge.Crowdsourced examples have the largest overall impact; head words particularly help ultra-fine labels, while entity linking helps fine labels.
- Analysis: The model generates accurate general types and diverse labels, but gold annotations omit many potentially correct labels, lowering measured precision.About half of the apparent precision errors belong to this missing-label category; other errors include co-hyponyms and context-unsupported types.
- Analysis: 28% of examples belong to cases where the model predicts only general types, contributing to low fine-grained recall.Even correct fine-grained predictions often omit a complete set of related labels.
- Limitations: Incomplete-label evaluation and expanding label coverage remain identified areas for future work.The model can also predict incongruous type pairs, motivating joint set prediction rather than independent label prediction.
6 Improving Existing Fine-Grained NER with Better Distant Supervision
The paper augments traditional knowledge-base supervision for OntoNotes with Wikipedia definitions and head-word supervision. This combination improves existing fine-grained entity typing performance, with head words producing especially broad gains across metrics.
- Training Data: The OntoNotes training data supplements knowledge-base supervision with Wikipedia definitions and head-word supervision.A manual mapping converts a noun from the natural-language vocabulary to each formal OntoNotes type.
- Experiment Setup: The evaluation reports macro- and micro-averaged F1 and exact-set-match accuracy.These metrics follow prior work on the dataset.
- Results: The combined model and training data achieve a clear improvement over prior work and set a new state-of-the-art result on OntoNotes.The evaluated model uses all of the paper’s distant-supervision sources.
- Ablation Study: Every added supervision source improves performance for both the AttentiveNER model and the paper’s model.Head-word supervision is the prime source of improvement, increasing performance by about 10% across all metrics.
- Evaluation Caveat: The reported comparison excludes a system whose 72.98 micro F1 came from a different, unreleased test set.The paper therefore does not compare that result directly with its test-set results.
7 Related Work
Related work spans fine-grained typing across sentence, corpus, and lexicon contexts, using ontologies derived from resources such as Wikipedia, Freebase, and WordNet. This paper differs by using open noun-phrase types, covering all noun phrases, conditioning on whole sentences, and pooling noisy supervision across granularities.
- Entity Typing Settings: Prior entity-typing work studies sentence-level, corpus-level, and lexicon-level prediction settings.The paper focuses on mentions in specific sentences.
- Fine-Grained Ontologies: Recent fine-grained ontologies use Wikipedia categories, Freebase types, or WordNet senses, but commonly focus on named entities and distant supervision.The paper instead uses frequent noun phrases that depict types and extends coverage beyond named entities.
- Contextual Typing: Unlike selectional-preference methods that focus on predicates, this task conditions type inference on the entire sentence.This enables more nuanced context-sensitive types for arguments.
- Noisy Supervision: The multitask objective pools different distant-supervision sources across levels of granularity to handle noisy supervision.This contrasts with prior approaches using multi-instance multi-label learning or custom losses.
8 Conclusion
Virtually unrestricted types expand knowledge-base training with Wikipedia-definition and naturally occurring head-word supervision. These sources improve performance on both the new dataset and an existing fine-grained entity-typing benchmark, establishing baselines for future work.
- Conclusion: Virtually unrestricted types expand knowledge-base training with Wikipedia-definition and naturally occurring head-word supervision.The new supervision sources improve performance on the new dataset and an existing fine-grained entity-typing benchmark.
- Conclusion: The results establish the first performance levels for the new evaluation dataset and suggest that it supports significant future work.