PaperlayerView exact revision on arXiv

Source-linked AI summary

Hierarchical Losses and New Resources for Fine-grained Entity Typing and Linking

Shikhar Murty*, Patrick Verga*, Luke Vilnis, Irena Radovanovic, Andrew McCallum

arXiv:1807.05127v1cs.CL

TL;DR

The paper tackles entity linking and fine-grained typing methods that treat labels as flat despite rich ontology hierarchies and limited shallow benchmarks. It models hierarchical structure with real and complex bilinear mappings and introduces MedMentions and TypeNet. Hierarchy-aware training improves results across all three datasets, including 6% relative accuracy on MedMentions, 29% relative MAP on TypeNet, and state-of-the-art FIGER performance without hand-crafted features.

  • Problem

    Existing entity linking and typing approaches largely ignore rich entity and type hierarchies, while available benchmarks are small and shallow.

    Source: §1 · p. 1

  • Method

    The paper integrates hierarchy information into neural entity and type embeddings using real and complex bilinear mappings and hierarchy-aware losses, alongside two annotated datasets.

    Source: §1 · pp. 1–2, §5 · p. 5, §5.1 · p. 5

  • Results

    Hierarchy-aware training improves all three datasets, yielding 6% relative MedMentions accuracy gain, 29% relative TypeNet MAP gain, and state-of-the-art FIGER performance without hand-crafted features.

    Source: §1 · p. 2, §8 · p. 9

  • Takeaways & Limitations

    Explicitly modeling hierarchical information improves entity typing and linking across shallow and deep ontologies while enabling evaluation on richer resources.

    Source: §1 · p. 2, §8 · p. 9

    The paper leaves improved entity-linking candidate-generation recall and alternative hierarchical embedding methods for future work.

    Source: §8 · p. 9

Abstract

from arXiv · show

Extraction from raw text to a knowledge base of entities and fine-grained types is often cast as prediction into a flat set of entity and type labels, neglecting the rich hierarchies over types and entities contained in curated ontologies. Previous attempts to incorporate hierarchical structure have yielded little benefit and are restricted to shallow ontologies. This paper presents new methods using real and complex bilinear mappings for integrating hierarchical information, yielding substantial improvement over flat predictions in entity linking and fine-grained entity typing, and achieving new state-of-the-art results for end-to-end models on the benchmark FIGER dataset. We also present two new human-annotated datasets containing wide and deep hierarchies which we will release to the community to encourage further research in this direction: MedMentions, a collection of PubMed abstracts in which 246k mentions have been mapped to the massive UMLS ontology; and TypeNet, which aligns Freebase types with the WordNet hierarchy to obtain nearly 2k entity types. In experiments on all three datasets we show substantial gains from hierarchy-aware training.

1 Introduction

The paper addresses the limitations of flat entity and type prediction by integrating ontology hierarchies into neural models and introducing larger, deeper evaluation resources. Hierarchy-aware training improves entity linking and typing across three datasets, including FIGER.

  • Flat prediction methods largely ignore entity and type hierarchies, while existing benchmarks such as FIGER contain only 113 types across two levels.Source: §1 · p. 1
  • Hierarchy-aware losses integrate ontology structure into entity and type embeddings, improving statistical efficiency for similar concepts and rarer types.Source: §1 · p. 1
  • MedMentions and TypeNet provide human-annotated resources with wide and deep hierarchies for further research in hierarchical entity linking and typing.Source: §1 · pp. 1–2
  • 6% relative accuracy gain is achieved for entity linking on MedMentions, while TypeNet entity typing gains 29% relative MAP from hierarchy-aware training.Source: §1 · p. 2
  • The approach matches state-of-the-art FIGER results without hand-crafted features, while improving performance across all three datasets.Source: §1 · p. 2

2 New Corpora and Ontologies

The paper introduces MedMentions and TypeNet to address the scarcity of richly annotated resources with wide and deep entity or type hierarchies. MedMentions links biomedical mentions to UMLS, while TypeNet aligns Freebase types with WordNet.

2.1 MedMentions

  • UMLS contains over 3.5 million concepts in a curated hierarchy and does not distinguish between entities and types.Source: §2.1 · p. 2
  • MedMentions contains 246,000 exhaustively annotated PubMed mention spans linked to UMLS concepts, whose annotated concepts average depth 14.4 and reach depth 43.Source: §2.1 · p. 2
  • MedMentions provides an order of magnitude more annotations than comparable biological linking datasets and covers diverse entity types rather than only narrow biomedical categories.Source: §2.1 · p. 2

2.2 TypeNet

  • TypeNet2 is a new dataset of hierarchical entity types designed for extremely fine-grained entity typing.Source: §2.2 · p. 2
  • TypeNet is built by manually aligning sufficiently frequent Freebase types to WordNet synsets using parent-of, child-of, or equivalence relations.Source: §2.2 · p. 3
  • Two expert annotators resolved alignment conflicts, producing a directed acyclic hierarchy that also incorporates verified Freebase-to-Freebase links.Source: §2.2 · p. 3

3 Model

The model represents mentions using both sentence context and mention-surface semantics, then predicts entity or type labels from the resulting vector. Its encoder marks the target span with relative positions, applies a CNN and max pooling to the sentence, and combines the contextual and global mention representations through a two-layer network.

3.1 Background: Entity Typing and Linking

  • Mention-level and entity-level tasks classify textual mentions or bags of mentions with entity and type labels.Entity-level typing aggregates mentions linked to the same entity, whereas mention-level typing labels an individual sentence.Source: §3.1 · p. 3

3.2 Mention Encoder

  • The encoder combines averaged mention-string embeddings with a CNN-based context representation to produce the final mention vector.The mention-string embeddings capture context-independent semantics, while the CNN encodes context-dependent information before a two-layer feed-forward network.Source: §3.2 · pp. 3–4, §3.2.2 · p. 4

3.2.1 Token Representation

  • Relative position embeddings distinguish the target mention from surrounding tokens, assigning position 0 inside the span and signed distances outside it.Each token representation concatenates a word embedding with a learned position embedding relative to the target entity.Source: §3.2.1 · p. 4

3.2.2 Sentence Representation

  • A single-layer CNN with tanh nonlinearity and pointwise max pooling converts the token sequence into a context embedding.The experiments use CNN filters with width w = 5.Source: §3.2.2 · p. 4
  • The encoder also averages word embeddings within the mention span, concatenates this global representation with the CNN output, and applies a two-layer tanh network.This preserves both mention-surface information and sentence-level context in mF.Source: §3.2.2 · p. 4, Figure 1 · p. 4

4 Training

Training uses task-specific objectives for mention typing, entity-level typing, and entity linking. The entity-level model addresses noisy distant supervision with bag-level aggregation, while linking combines contextual similarity with candidate-string similarity.

4.1 Mention-Level Typing

  • Mention-level typing scores every type independently and optimizes a multi-label binary cross-entropy objective against the gold type vector.A type is positive when it belongs to the mention’s gold type set.Source: §4.1 · p. 4

4.2 Entity-Level Typing

  • Entity-level typing uses bags of mentions because distant supervision noisily assigns every entity type to every mention.The bag is labeled with the entity’s type vector rather than treating each noisy mention label as independently reliable.Source: §4.2 · pp. 4–5
  • Entity-level predictions encode sampled mentions independently, pool their logits with element-wise LogSumExp, and optimize multi-label binary cross entropy at the bag level.For each entity, the model subsamples k mentions before producing the pooled prediction.Source: §4.2 · p. 5

4.3 Entity Linking

  • Entity linking generates 100 candidate entities per mention and combines contextual embedding similarity with string similarity through a learned linear combination.The model predicts the candidate with the maximum final similarity and trains with multinomial cross entropy over candidates.Source: §4.3 · p. 5

5 Encoding Hierarchies

The paper adds hierarchy-aware training to otherwise flat entity and type prediction by modeling parent-child structure in the embedding space. It compares real and complex bilinear maps and trains them with positive and sampled negative hierarchy links.

5 Encoding Hierarchies

  • Hierarchy-aware training adds a structural loss to flat typing and linking objectives, modeling entity or type relationships with real or complex bilinear maps.The hierarchy is treated as a knowledge graph with a single hypernym or IS-A relation.Source: §5 · p. 5, §5.2 · p. 6

5.1 Hierarchical Structure Models

  • The real bilinear model scores a child-parent link with a learned nondiagonal matrix and reuses type embeddings on both sides of the relation.Its formulation is equivalent to RESCAL with one IS-A relation type, and the map is inserted before type or entity scoring.Source: §5.1 · p. 5
  • The complex bilinear model uses complex type embeddings, diagonal complex relation matrices, Hermitian inner products, and a real-valued final score.Complex-valued mention representations are produced by separate affine transformations for real and imaginary components.Source: §5.1 · pp. 5–6

5.2 Training with Hierarchies

  • Hierarchy training samples positive child-parent links and n negative links, then uses binary cross entropy to score positives high and negatives low.The structural-loss weighting parameter γ is selected from {0.1, 0.5, 0.8, 1, 2.0, 4.0}.Source: §5.2 · p. 6
  • The complex hierarchical loss is not itself technically bilinear after the complex transformation, because that step commutes with addition but not complex multiplication.The paper nevertheless calls the model bilinear for ease of exposition.Source: §5.2 · p. 6

6 Experiments

The experiments evaluate hierarchy-aware models across FIGER mention typing, TypeNet entity typing, and MedMentions entity linking. Hierarchical losses and transitive information improve performance, with gains varying by dataset and model.

6 Experiments

  • The study compares mention-level FIGER typing, entity-level Wikipedia typing with TypeNet, and MedMentions linking to UMLS.TypeNet evaluation uses MAP over Freebase types and aggregates mention-level predictions with LogSumExp pooling.Source: §6 · p. 6, §6.3 · p. 7

6.1 Models

  • The evaluated models include flat CNN baselines, CNN+Complex, transitive-closure training, and explicit real- or complex-valued hierarchy losses.Hierarchy models add an explicit hierarchical loss, while transitive models add ancestor labels through the transitive closure.Source: §6.1 · p. 6

6.2.1 Results

  • 2.5 points of accuracy lift the CNN+Complex FIGER model to the overall state of the art without hand-crafted features.The hierarchy also helps predict lower-frequency types, paralleling the role of hand-crafted features in the comparison.Source: §6.2.1 · pp. 6–7, Table 5 · p. 7

6.3.1 Results

  • On TypeNet, explicit hierarchy loss and transitive closure each improve both CNN variants, with their combination producing further gains.The basic CNN improves more than CNN+Complex, and the authors suggest complex embeddings may be harder to optimize.Source: §6.3.1 · p. 7
  • Across the reported cases, baseline models without hierarchical information—neither transitive closure nor hierarchy loss—perform ~9 MAP worse, demonstrating the benefits of incorporating structure information.Similar trends are observed with ~150,000 examples, or about 5% of the total data.Source: §6.3.1 · p. 7

6.4.1 Results

  • 6% relative error reduction on MedMentions follows from incorporating structure information, although transitive closure is omitted because candidate generation makes it difficult.The MedMentions setup uses an imperfect alias table, with maximum recall of 81.82%.Source: Table 7 · p. 7, §6.4.1 · p. 8
  • MedMentions qualitative examples show hierarchy information disambiguating rare entities and avoiding predictions that are more specific than the correct entity.For this task, complex hierarchical loss gives a significant boost whereas the real-valued bilinear model does not.Source: §6.4.1 · p. 8, Table 8 · p. 8

7 Related Work

The paper connects hierarchical entity typing and linking to prior work on knowledge-base inference, flat entity prediction, and learned concept hierarchies. It positions its contribution as combining structural learning with new resources for large-scale ontology mapping.

7 Related Work

  • The hierarchy-aware loss builds on knowledge-base inference and embedding work, including RESCAL- and ComplEx-related research [Jain et al. (2018); Trouillon et al. (2016); Nickel et al. (2011)].Prior work also explored combining knowledge-base completion with hierarchical structure and hierarchical losses for text classification.Source: §7 · p. 8
  • Earlier fine-grained typing approaches using shallow hierarchies produced no significant or consistent improvements, motivating the paper’s structural approach [Gillick et al., 2014; Shimaoka et al., 2017].Related work also includes long-standing flat entity linking and typing across multiple granularity levels.Source: §7 · p. 8
  • Prior resources and methods derive type hierarchies from Yago and WordNet, including manually crafted rules and patterns for matching types to WordNet synsets [Yosef et al., 2012; Del Corro et al. (2015)].These lines of work provide related approaches to constructing hierarchical type systems.Source: §7 · pp. 8–9
  • Recent research also unifies entity linking with fine-grained type prediction and learns hierarchical or transitive relations between concepts [Vilnis and McCallum, 2015; Vendrov et al., 2016; Nickel and Kiela, 2017].Source: §7 · p. 9
  • The work contributes datasets and models for jointly modeling large hierarchical ontologies while mapping textual mentions into fine-grained entity and type spaces.This contribution draws together research threads in information extraction, knowledge-base population, and completion.Source: §7 · pp. 8–9

8 Conclusion

The paper concludes that explicit hierarchical modeling improves entity typing and linking across three challenging datasets. It also contributes MedMentions and TypeNet as large, human-annotated resources for future research.

8 Conclusion

  • Explicitly modeling hierarchy increases performance across entity typing and linking experiments on three challenging datasets.Source: §8 · p. 9
  • MedMentions provides 246k PubMed mentions linked to UMLS, while TypeNet supplies a hierarchical fine-grained type set an order of magnitude larger and deeper than previous datasets.Source: §8 · p. 9
  • Future work includes alternative hierarchical embeddings and improved entity-linking candidate-generation recall.The paper specifically names Box and Poincaré embeddings as possible alternatives.Source: §8 · p. 9

A.1 TypeNet Construction

The TypeNet construction example illustrates how Freebase types are aligned to WordNet synsets using candidate generation and manual selection.

A.1 TypeNet Construction

  • TypeNet annotators link a Freebase type such as musical chord to candidate WordNet synsets generated by substring matching.The correctly aligned synset in the example is chord.n.02.Source: Table 9 · p. 13

A.2 Model Implementation Details

The experiments use fixed pretrained word embeddings, tuned regularization and optimization, and hierarchy-aware training with dataset-specific inference and candidate-generation procedures. The Wikipedia and UMLS setups define the data splits, representations, and candidate sets used for evaluation.

A.2 Model Implementation Details

  • Fixed 300-dimensional pretrained word vectors are combined with 300-dimensional Glorot-initialized type and entity vectors, while hierarchical training samples 16–256 negative links.The pretrained embeddings remain fixed during training.Source: §A.2 · p. 13
  • Dropout, L2 regularization, Adam optimization, grid-search hyperparameter tuning, and development-set early stopping provide the common training procedure.Dropout rates are selected from {0.5, 0.75, 0.8}, L2 coefficients from {1e-5, 5e-5, 1e-4}, and the learning rate is 0.001.Source: §A.2 · p. 13

A.3 FIGER Implementation Details

  • FIGER models train with mention typing loss plus a tuned-weight hierarchical loss, then assign the highest-probability type and add types whose probabilities exceed 0.5.Inference follows the procedure of Shimaoka et al. (2017).Source: §A.3 · p. 13

A.4 Wikipedia Data and Implementation Details

  • The Wikipedia dataset contains 344,246 entities mapped to 1,081 TypeNet types, uses 10–50-token sentences and entity bags, and splits entities 90-5-5 into train, development, and test sets.Training samples bags of 10 mentions, whereas testing classifies bags of 20 mentions.Source: §A.4 · p. 13

A.5.1 Candidate Generation Details

  • UMLS candidate generation uses normalized TFIDF character ngram vectors with ngrams of size 1–5, retains 100,000 features, and selects the 100 most similar entities by cosine similarity.The same preprocessing and candidate-selection procedure is specified for each mention and canonical UMLS entity string.Source: §A.5.1 · p. 13, §A.5 · p. 13

Related papers

Loading 1807.05127v1…