PaperlayerView exact revision on arXiv

Source-linked AI summary

Investigating Entity Knowledge in BERT with Simple Neural End-To-End Entity Linking

Samuel Broscheit

arXiv:2003.05473v1cs.CLcs.LG

TL;DR

Entity linking combines mention detection, candidate generation, and disambiguation, raising questions about whether BERT can learn these steps jointly and how much entity knowledge it contains. The paper studies a simplified BERT-based approach and finds additional entity knowledge helps entity linking, while most tested downstream benchmarks do not benefit from it.

  • Problem

    The study asks whether entity-linking steps can be learned jointly with BERT, how much entity knowledge pretrained BERT contains, and whether additional entity knowledge helps downstream tasks.

    Source: §1 · p. 2

  • Method

    The paper studies a simplified BERT-based entity-linking model trained from Wikipedia-derived data, without a pipeline or heuristics.

    Source: §3 · p. 3, §6 · p. 8

  • Results

    BERT+Entity learns additional entity knowledge for entity linking, but almost none of the tested downstream tasks benefit; GLUE RTE improves 2%.

    Source: §1 · p. 2, §4.3.2 · p. 6, §6 · p. 8

  • Takeaways & Limitations

    The authors identify whether downstream tasks require entity knowledge as an open question for future research.

    Source: §1 · p. 2, §6 · p. 8

    Wikipedia link annotations are incomplete, which the authors hypothesize can be detrimental during training.

    Source: §3 · p. 3

Abstract

from arXiv · show

A typical architecture for end-to-end entity linking systems consists of three steps: mention detection, candidate generation and entity disambiguation. In this study we investigate the following questions: (a) Can all those steps be learned jointly with a model for contextualized text-representations, i.e. BERT (Devlin et al., 2019)? (b) How much entity knowledge is already contained in pretrained BERT? (c) Does additional entity knowledge improve BERT's performance in downstream tasks? To this end, we propose an extreme simplification of the entity linking setup that works surprisingly well: simply cast it as a per token classification over the entire entity vocabulary (over 700K classes in our case). We show on an entity linking benchmark that (i) this model improves the entity representations over plain BERT, (ii) that it outperforms entity linking architectures that optimize the tasks separately and (iii) that it only comes second to the current state-of-the-art that does mention detection and entity disambiguation jointly. Additionally, we investigate the usefulness of entity-aware token-representations in the text-understanding benchmark GLUE, as well as the question answering benchmarks SQUAD V2 and SWAG and also the EN-DE WMT14 machine translation benchmark. To our surprise, we find that most of those benchmarks do not benefit from additional entity knowledge, except for a task with very small training data, the RTE task in GLUE, which improves by 2%.

1 Introduction

The introduction asks whether BERT can jointly perform entity linking and whether entity knowledge is already present in BERT or helps downstream tasks. It presents per-token classification as a unified approach and reports gains in entity linking but little downstream benefit.

1 Introduction

  • Entity linking typically comprises mention detection, candidate generation, and entity disambiguation; candidate generation can limit coverage when the correct entity is absent from the candidates.Mention detection also determines the upper bound of linkable mentions.Source: §1 · p. 1
  • BERT+Entity casts mention detection, candidate generation, and entity disambiguation as per-token classification over the entity vocabulary, without mention-span supervision.The vocabulary comprises the 700K most frequent English Wikipedia entities; validation analysis attributes only 3% of errors purely to span errors, with most involving wrong Nil predictions.Source: §1 · p. 2
  • Frozen-BERT+Entity performs 6% below BERT+Entity, indicating that the proposed model learns entity knowledge beyond what pretrained BERT already contains.Frozen-BERT+Entity freezes BERT and trains only the entity classifier.Source: §1 · p. 2
  • Additional entity knowledge does not help GLUE, SQUAD V2, SWAG, or EN-DE WMT14 overall; RTE is the exception, improving 2%.RTE has just 0.5–2% of the training data of the two larger GLUE natural-language-inference datasets.Source: §1 · p. 2
  • The study proposes a fully neural model for all three entity-linking steps and identifies a lack of tasks that evaluate additional entity knowledge in pretrained language models.The authors describe token classification as a viable option for entity linking.Source: §1 · p. 2

2 Related Work

Prior work jointly modeled some or all entity-linking stages, while neural language-model representations motivated studying entity knowledge within BERT. The paper positions its work as examining how much entity information BERT contains and whether it can be improved.

2 Related Work

  • Durrett and Klein (2014) jointly modeled interactions among mention detection, candidate generation, and entity disambiguation, finding the joint objective beneficial to each task; Nguyen et al. (2016) jointly modeled mention detection and entity disambiguation with a graphical model.Kolitsas et al. (2018) overgenerate mentions, prune them with a mention-entity dictionary, and reason over the remaining mentions.Source: §2 · p. 3
  • Durrett and Klein (2014) jointly modeled MD, CG, and ED; they found the joint objective beneficial, with each task improving, and argued the tasks have no natural order and should interact freely.Yamada et al. (2016, 2017) first investigated neural text representations and entity linking, but their approach was limited to ED.Source: §2 · p. 3
  • Pretrained language models provide deep contextualized text representations, and prior work used entity knowledge in BERT pretraining to improve relation extraction (Zhang et al., 2019).This study examines what entity information is identifiable in BERT and whether it can be improved.Source: §2 · p. 3

3 End-To-End Neural Entity-Linking

BERT+Entity extends BERT for entity linking, with training data and annotations derived from Wikipedia links. The training scheme addresses incomplete link annotations by augmenting matches and adjusting Nil-link counts.

3 End-To-End Neural Entity-Linking

  • Wikipedia-derived link annotations may be incomplete, and the authors hypothesize that this incompleteness can be detrimental during training.The model is trained on Wikipedia text and aims to evaluate candidate generation with a large entity vocabulary.Source: §3 · p. 3

3.1 Model

  • BERT+Entity adds an output classifier that computes, for each contextualized token, probabilities of linking to entries in the entity vocabulary.The classifier uses BERT-base-uncased weights; each entity has a classification-layer representation, and the link probability is computed with a sigmoid.Source: §3.1 · p. 3

3.2 Training Data

  • The training data uses Wikipedia links as annotations and a Trie-based matcher to label occurrences of collected linkable strings, including unlinked mentions in selected fragments.To counter skew from incomplete annotations, the method selects fragments with a minimum count of annotated links and discounts Nil counts using the average Nil-link probability for the 1,000 most frequent entities.Source: §3.2 · p. 4

4 Entity Linking Experiments

The experiments evaluate BERT+Entity on entity linking and downstream tasks, including how training and data choices affect its performance. The section compares it with BERT baselines and prior entity-linking systems, then discusses error patterns and entity knowledge.

4.1 Data

  • Setting I uses 700K entities and 110-token fragments; Setting II uses 500K entities plus approximately 1,000 benchmark entities and 250-token fragments.The settings yield 8.8M and 2.4M training instances, respectively; CoNLL03/AIDA supplies training, validation, and test documents.Source: §4.1 · pp. 4–5

4.2 Training

  • To limit memory use, training samples true entities and high-scoring predicted entities, then computes token predictions over that batch-specific subset.The entity vocabulary is too large for loss computation over all entities; the training schedule first freezes BERT and then fine-tunes it with the entity classifier.Source: §4.2 · p. 5

4.3 Performance Metrics

  • Strong match requires every token in a gold span to be correct, weak match requires at least one, and strong ED Precision@1 excludes Nil predictions.The reported metrics include Micro InKB precision, recall, and F1.Source: §4.3 · p. 5

4.3.1 Results

  • The BERT baseline trains its entity classifier from scratch on CoNLL03/AIDA, while Frozen-BERT+Entity keeps BERT fixed and trains only the classifier.The BERT baseline cannot generalize to entities absent from its training data; BERT+Entity trains both BERT and the classifier on Wikipedia.Source: §4.3.1 · p. 6

4.3.2 Discussion

  • BERT+Entity scores 6%-10% above Frozen-BERT+Entity, while Frozen-BERT+Entity scores 13%-16% above BERT, depending on score and dataset.Setting II comes close to Kolitsas et al. (2018); the authors conjecture that capping examples per entity allowed more training epochs and may have improved less-frequent entity representations.Source: §4.3.2 · p. 6
  • The downstream experiments cover GLUE, SQuAD V2, SWAG, and WMT14 EN-DE; only dev-set results are reported for SQuAD V2 and SWAG.Table 3 reports experiments using BERT+Entity trained in Setting I.Source: Table 3 · p. 7

4.3.1 Results

4.3.2 Discussion

  • The largest sampled error source was predicting Nil instead of an entity; insufficient context also caused errors, while wrong spans accounted for 3%.These findings motivated the follow-up Setting II experiments.Source: Table 2 · p. 6, §4.3.2 · p. 7

5 Downstream Tasks Experiments

Across GLUE, SQUAD V2, SWAG, and EN-DE WMT14, additional entity knowledge generally did not help; RTE was the exception. The section evaluates ensemble models and fixed-encoder sequence-to-sequence models under specified training settings.

5.3 Results

  • Only RTE benefits from additional entity knowledge; the other evaluated tasks show no advantage, and the same outcomes hold in Settings I and II.The authors leave entity overlap, entity scarcity, and whether tasks require entity knowledge as open explanations.Source: §5 · p. 7, §5.3 · p. 8

5.1 Model

  • For GLUE, SQUAD V2, and SWAG, the experiments compare concatenated BERT+Entity and BERT ensembles; for EN-DE WMT14, they use BERT or BERT+Entity encoders with a Transformer decoder.The sequence-to-sequence systems adapt fairseq’s Pytorch Seq2Seq Transformer implementation.Source: §5.1 · p. 7

5.2 Training

  • GLUE, SQUAD V2, and SWAG ensembles were trained for three epochs; the machine-translation models used four epochs and fixed encoders for the first epoch.The translation setup used Adam with a 5e-5 learning rate, batches capped at 1,000 tokens, gradient clipping at 0.1, dropout 0.2, and label smoothing 0.1.Source: §5.2 · p. 8

6 Conclusion

The simplified entity-linking approach learns additional entity knowledge that helps entity linking, but almost none of the evaluated downstream tasks require it. The authors identify the remaining entity-linking gap to state of the art as a hypothesis for future scaling, not a demonstrated result.

6 Conclusion

  • The approach learns additional entity knowledge that helps entity linking, while almost none of the downstream tasks require it.The authors describe the latter finding as an open question for future research.Source: §6 · p. 8
  • The model performs entity linking without a pipeline or heuristics, unlike prior approaches.Source: §6 · p. 8
  • The authors hypothesize that larger hardware capacity, model size, and effective training time could close the gap to current state of the art.Source: §6 · p. 8
Loading 2003.05473v1…