Context-Dependent Fine-Grained Entity Type Tagging

• Fine-grained labels can substantially improve relation extraction and possibly other downstream tasks compared with the small label sets used by most tagging systems.Source: §1 · p. 1
• Knowledge-base supervision assigns multiple entity types that may not apply to a mention’s local context, creating spurious training and evaluation labels.Existing systems derive examples automatically from resolved Wikipedia entities and their Freebase or YAGO types.Source: §1 · pp. 1–2
• Context-dependent fine-type tagging restricts acceptable labels to those relevant or deducible from the local sentence, paragraph, or document.Source: §1 · p. 2
• Label-pruning heuristics reduce spurious labels in automatically generated training examples and improve performance on manually annotated data.Source: §1 · p. 2
• The paper introduces 12,017 manually annotated mentions from the OntoNotes test corpus as resources for research and experimental comparison.Source: §1 · p. 2

2 Fine-grained type labels and manual annotations

• The taxonomy organizes Freebase-derived labels in a hierarchy whose top-level categories align with common coarse entity types and whose paths encode IS-A relations.The hierarchy supports label consistency, domain knowledge, and top-down inference; ambiguous mappings back off to a common parent.Source: §2 · p. 2, Figure 1 · p. 3

2.1 Manual annotations

• Annotations include only types deducible from the sentence or paragraph, with annotators backing off to a parent type when the appropriate specificity is uncertain.Mention text itself can support common-knowledge labels such as location/city for “San Francisco.”Source: §2.1 · p. 2
• The corpus’s taxonomy contains types at three hierarchy levels; level-1 labels are ten times as common as level-3 labels because partial paths are allowed and some level-3 labels rarely occur in training data.Many level-2 types have no sub-types.Source: §2.1 · p. 3, Figure 1 · p. 3
• Six annotators labeled each document, retaining labels supported by at least two annotators and pruning about one quarter of labels.Agreement pruning reduced pairwise specificity disagreements from 3900 to 1303 and type disagreements from 3700 to 774.Source: §2.1 · p. 3
• Annotator precision was 92% at depth 2 and 89% at depth 3, although agreement decreased for deeper hierarchy types.These results support the task’s reasonable definition despite disagreements over specificity and type interpretation.Source: §2.1 · p. 3
• Automatically obtained Freebase labels had 80% overall recall but only 50% precision, indicating that many generated types were unrelated to mention contexts.Source: §2.1 · pp. 3–4

3.1 Assembling training data

• A 133,000-document news corpus is processed with NLP tools for mention extraction, entity resolution, and mapping resolved entities to Freebase-derived types.The pipeline separates entity mention identification from type tagging and uses alias-based linking to Freebase profiles.Source: §3.1 · p. 4

Sibling pruning

• Sibling pruning replaces multiple sibling types for one entity with their parent type because several sibling types are considered unlikely to be relevant in one context.This can remove correct labels, but less common entities with fewer Freebase types are treated as better training sources.Source: Sibling pruning · pp. 4–5

Coarse type pruning

• Coarse type pruning removes fine-grained labels that conflict with a classifier over person, location, organization, and other.The heuristic encourages labels to occupy one hierarchy subtree and can resolve location–organization conflicts.Source: Coarse type pruning · p. 5

Minimum count pruning

• Minimum-count pruning removes document types appearing fewer than twice, assuming context-relevant types recur across multiple mentions.For example, organization/sports-team may recur in a sports article.Source: Minimum count pruning · p. 5
• The heuristics reduce training examples and interact because earlier pruning changes the labels available to later heuristics.They are applied in the order introduced.Source: Minimum count pruning · p. 5

4.1 Feature extraction

• Feature extraction combines lexical and syntactic indicators with eight-topic document features and word clusters to represent mention context.The same feature representation is used across taxonomy levels, although deeper types may benefit more from contextual features.Source: §4.1 · p. 5, Table 4 · p. 6

4.2 Models and inference

• Hierarchical classification treats the type taxonomy as structured multilabel output and supports flat, local, and local-per-parent model families.The paper explores maximum-entropy logistic and softmax regression classifiers for flat and local approaches.Source: §4.2 · pp. 5–6

4.2.1 Local classifiers

• Local classifiers train one L2-regularized binary logistic model per type, treating descendants as positives and comparing three definitions of negative examples.At inference, the models use independent, conditional, or hierarchy-constraint-marginalizing strategies.Source: §4.2.1 · p. 6
• Marginalized inference sums probabilities over valid hierarchy paths, then assigns labels whose refined probabilities exceed a threshold.The paper enumerates paths directly and notes that larger taxonomies could use the sum-product algorithm.Source: §4.2.1 · p. 6

4.2.2 Flat classifier

• The flat softmax classifier converts each multilabel training instance into multiple single-label examples and thresholds label probabilities at inference.This allows multiple labels to be assigned despite the single-label training objective.Source: §4.2.2 · pp. 6–7

5 Experiments

• The evaluation reports precision, recall, F-score, and precision-recall AUC overall and separately at each taxonomy level.Level-specific reporting addresses the dominance of coarse level-1 types in aggregate metrics.Source: §5 · p. 7, Table 6 · p. 7
• The test set contains 61 documents and 11,197 named or nominal non-pronominal mentions, with one development-set threshold chosen to maximize overall F-score.The authors note that label-specific thresholds might improve performance because individual labels show wide precision-recall variation.Source: §5 · p. 7

5.1 Classifiers and inference

• Among local-classifier configurations, same-depth negative examples combined with marginalizing hierarchy constraints produce the best results.The comparison covers three negative-example strategies and three inference methods.Source: §5.1 · p. 7
• The local classifier reaches 69.3% AUC versus 63.7% for the flat classifier, with the advantage especially pronounced at deeper taxonomy levels.The comparison uses identical features and total numbers of model parameters for both approaches.Source: §5.1 · p. 7, Table 6 · p. 7

5.2 Distant supervision heuristics

• Label pruning improves overall F1 by 11.3% and AUC by 7.2%, with coarse pruning particularly important.These results use local classifiers, same-depth negative examples, and inference that marginalizes hierarchy constraints.Source: §5.2 · p. 7

Conclusions

• Context-dependent tagging requires labels to be deducible from local context, reducing mismatch between automatically generated training data and manually annotated gold data.The approach addresses spurious knowledge-base-derived labels through distant-supervision pruning heuristics.Source: §6 · pp. 7–8

Experiments

• Table 5 compares negative-example strategies for local classification with inference schemes, using all pruning heuristics in both experiments.The table’s best settings are identified by comparing the left-side training strategies and right-side inference methods.Source: Table 5 · p. 8
• Label-pruning heuristics each improve system performance over the baseline, while applying them together produces the largest improvement, particularly in precision.Source: Table 7 · p. 8

Resources

• The paper introduces 12,017 manually annotated mentions in 77 OntoNotes news documents with 17,704 type labels as a basis for experimental comparison.These resources are intended to support future research and meaningful comparisons on the task.Source: §6 · p. 8

Limitations and future work

• The experiments expose difficulties with very specific labels when relatively few examples are available.The paper suggests richer hierarchical models, joint document-level mention labeling, and additional label constraints as future directions.Source: §6 · p. 8

Conclusions

• The problem formulation, training heuristics, and new resources are intended to provide a meaningful framework for future research.Source: §6 · p. 8