PaperlayerView exact revision on arXiv

Source-linked AI summary

E-BERT: Efficient-Yet-Effective Entity Embeddings for BERT

Nina Poerner, Ulli Waltinger, Hinrich Schütze

arXiv:1911.03681v2cs.CL

TL;DR

BERT can rely on entity-name surface forms rather than factual knowledge, motivating a method that injects entity information more directly. E-BERT aligns Wikipedia2Vec entity vectors with BERT’s wordpiece space and uses them without further encoder pretraining, outperforming baselines across QA, relation classification, and entity linking while reducing name-based reliance.

  • Problem

    LAMA may reward models for exploiting helpful entity names, making it difficult to distinguish factual knowledge from surface-form reasoning.

  • Method

    E-BERT aligns Wikipedia2Vec entity vectors with BERT’s wordpiece space and feeds them into the unchanged BERT encoder without additional pretraining.

  • Results

    E-BERT outperforms BERT and other baselines on unsupervised QA, relation classification, and entity linking, while E-BERT-replace drops by less than 1% across LAMA and LAMA-UHN.

  • Takeaways & Limitations

    E-BERT’s results indicate that entity vectors provide factual information complementary to entity names, while reducing reliance on helpful names.

  • Takeaways & Limitations

    Entity vectors are unavailable for 6% of LAMA questions and 10% of FewRel sentences, so those cases use standard BERT behavior.

Abstract

from arXiv · show

We present a novel way of injecting factual knowledge about entities into the pretrained BERT model (Devlin et al., 2019): We align Wikipedia2Vec entity vectors (Yamada et al., 2016) with BERT's native wordpiece vector space and use the aligned entity vectors as if they were wordpiece vectors. The resulting entity-enhanced version of BERT (called E-BERT) is similar in spirit to ERNIE (Zhang et al., 2019) and KnowBert (Peters et al., 2019), but it requires no expensive further pretraining of the BERT encoder. We evaluate E-BERT on unsupervised question answering (QA), supervised relation classification (RC) and entity linking (EL). On all three tasks, E-BERT outperforms BERT and other baselines. We also show quantitatively that the original BERT model is overly reliant on the surface form of entity names (e.g., guessing that someone with an Italian-sounding name speaks Italian), and that E-BERT mitigates this problem.

1 Introduction

E-BERT injects factual entity knowledge into BERT by aligning Wikipedia2Vec entity vectors with BERT’s wordpiece space, without changing or further pretraining the encoder. It improves performance across unsupervised QA, relation classification, and entity linking, while motivating LAMA-UHN to test reliance on entity-name surface forms.

  • Approach: E-BERT aligns Wikipedia2Vec entity vectors with BERT’s wordpiece vectors and feeds them into BERT without changing or further pretraining its encoder.This design contrasts with entity-enhanced models such as ERNIE and KnowBert, which require additional encoder pretraining.
  • Evaluation: E-BERT outperforms BERT, ERNIE, and KnowBert on the LAMA unsupervised QA benchmark.The paper evaluates factual knowledge retrieval without supervised task training.
  • Evaluation: LAMA-UHN removes questions with overly informative entity names to evaluate factual knowledge separately from name-based prediction.The benchmark targets cases where surface forms may provide misleading clues, such as inferring a person’s language from an Italian-sounding name.
  • Evaluation: E-BERT also outperforms original BERT and other baselines on supervised relation classification and entity linking.Aligned entity vectors serve as inputs for relation classification and as inputs and outputs for entity linking.
  • Resources: The paper releases LAMA-UHN and E-BERTBASE and E-BERTLARGE upon publication.

2 Related work

Related work establishes BERT as a pretrained masked language model and situates E-BERT among approaches that add factual or linguistic knowledge. E-BERT differs by adapting entity vectors to BERT rather than adding encoder components and further pretraining.

  • BERT: BERT is a Transformer pretrained as a masked language model that represents text with wordpiece embeddings processed by an encoder.Its masked-language-modeling head predicts masked wordpieces during pretraining and is typically replaced by a task-specific layer afterward.
  • Entity-enhanced BERT: ERNIE and KnowBert add encoder layers and require additional pretraining to integrate pretrained entity vectors into BERT.Their design adapts BERT to entity vectors rather than adapting entity vectors to BERT.
  • Knowledge-enhanced language models: KEPLER and K-Adapter use RoBERTa-based multi-task training on relation classification and knowledge-base completion rather than entity-vector representations.
  • Entity representations: Wikipedia2Vec embeds words and Wikipedia entities in a shared space using word-context, hyperlink-graph, and entity-to-word objectives.The shared space enables the same alignment strategy to transfer entity representations into BERT’s wordpiece space.
  • Unsupervised QA: Prior unsupervised QA work uses pretrained language models, but does not quantitatively distinguish factual knowledge from reliance on entity-name surface forms.

3 E-BERT

E-BERT maps Wikipedia2Vec entity vectors into BERT’s wordpiece space and feeds them through the unchanged BERT encoder. The section presents concatenation and replacement variants, with implementation designed to reuse standard BERT processing.

  • Vector alignment: E-BERT learns a linear mapping that transforms Wikipedia2Vec vectors to resemble BERT’s native wordpiece vectors.The mapping is fitted using items shared by the word and wordpiece vocabularies, then applied to entity vectors because Wikipedia2Vec places words and entities in one space.
  • E-BERT-concat: E-BERT-concat combines entity IDs with wordpieces using slash-separated strings, preserving both entity vectors and surface-form tokens.The resulting vectors are combined with position and segment embeddings before entering the unchanged BERT encoder.
  • Encoder integration: Both E-BERT variants feed aligned entity representations into BERT without changing or further pretraining the encoder.This is the central efficiency distinction from ERNIE and KnowBert.
  • E-BERT-replace: E-BERT-replace substitutes an entity’s surface form with its aligned entity vector as an ablation variant.This variant removes the entity-name wordpieces while retaining the entity representation.
  • Implementation: Wikipedia2Vec training uses a 2019 Wikipedia dump and produces an entity vocabulary of 2.7M entities.Training took approximately 6 hours on 32 CPUs, while fitting the alignment transformation was negligible and required no GPU.

4 Unsupervised QA

The unsupervised QA evaluation uses LAMA cloze questions to test factual knowledge without supervised fine-tuning, while examining whether BERT exploits informative entity names. E-BERT improves over BERT and other baselines, and its advantage is more robust when helpful names are removed.

  • LAMA benchmark: LAMA evaluates pretrained language models on factual cloze questions without supervised fine-tuning.The benchmark elicits an object from a knowledge-base triple and evaluates predictions against the object's surface form.
  • Baselines: The evaluation compares cased BERTBASE and BERTLARGE with entity-enhanced baselines including ERNIE and KnowBert.KnowBert has a smaller entity vocabulary, so results also consider questions whose gold subjects are in its vocabulary.
  • Name-based reasoning: BERT can exploit entity-name associations, such as inferring a person's native language from their name, rather than relying on factual knowledge.This makes original LAMA unable to distinguish memorized facts from reasoning about entity-name surface forms.
  • LAMA-UHN: LAMA-UHN removes questions with helpful entity names using string-match and person-name filters, creating a harder factual-knowledge evaluation.The string-match filter removes triples whose answers occur in subject names, while the person-name filter removes name-associated guesses.
  • Results: E-BERT-concatBASE sets a new state of the art on LAMA, while E-BERT-replace drops less than 1% from LAMA to LAMA-UHN compared with BERT's drop of about 8%.These results suggest E-BERT-replace relies more strongly on factual knowledge, while E-BERT-concat combines entity-name and entity-vector information.
  • Results: E-BERT-concat gains especially strongly over BERT, KnowBert, and ERNIE for mean Hits@k when k > 1.The result indicates a larger advantage in placing the correct answer near the top than in selecting it as the single top prediction.

5 Downstream tasks

E-BERT is applied to relation classification and entity linking by feeding aligned entity vectors into BERT, with entity linking additionally using them as outputs. Across both tasks, aligned vectors improve over original BERT and other baselines, while alignment is important for performance.

  • Downstream task setup: E-BERT evaluates aligned entity vectors on relation classification and entity linking, keeping the embedding layer fixed while fine-tuning the remaining encoder parameters.The experiments use the BERTBASE architecture.
  • Relation classification: In relation classification, E-BERT-concat performs better than original BERT and slightly better than ERNIE.E-BERT-replace also beats original BERT, indicating that aligned entity vectors can be more useful than entity names for this task.
  • Relation classification: The drop in F1 from E-BERT to Wikipedia2Vec-BERT shows the importance of vector space alignment.The non-aligned baselines feed Wikipedia2Vec vectors directly into BERT instead of aligned entity vectors.
  • Entity linking: Entity linking detects candidate spans, generates candidate entities with priors, and selects the correct candidate for each span.E-BERT-MLM uses an entity mask whose output distribution is over candidate entities plus a null entity for rejected spans.
  • Entity linking: Aligned entity vectors give E-BERT-MLM a strong initialization for predicting entities from context, producing better unsupervised performance, faster convergence, and higher final F1 than non-aligned baselines.The model iteratively replaces selected spans with decoded entity vectors, using up to three iterations because further iterations did not improve the development set.

6 Conclusion

E-BERT injects factual entity knowledge by aligning Wikipedia2Vec vectors with BERT’s wordpiece space, without changing the BERT encoder. It achieves strong results on unsupervised QA and supervised downstream tasks while exposing name-surface reliance in BERT.

  • Contribution: E-BERT aligns Wikipedia2Vec entity vectors with BERT’s wordpiece space and feeds them into BERT without changing the encoder.This avoids the additional encoder layers and expensive further pretraining used by ERNIE and KnowBert.
  • Question answering: E-BERT sets a new state of the art on the LAMA unsupervised QA benchmark.LAMA-UHN removes questions with helpful entity names to quantify reliance on surface forms.
  • Downstream tasks: E-BERT achieves competitive results relative to BERT and other baselines on supervised relation classification and entity linking.

Data

The paper evaluates factual knowledge on selected LAMA relations using downloaded data and reports relation-level results before and after applying LAMA-UHN heuristics.

  • Evaluation data: The evaluation uses the LAMA-T-REx and LAMA-Google-RE relations, which target factual knowledge.Table 9 reports individual-relation results and question counts before and after the LAMA-UHN heuristics.

Preprocessing

LAMA surface forms are mapped to entity IDs through Wikidata and then to corresponding English Wikipedia URLs using SPARQL queries.

  • Entity mapping: Surface forms are mapped to Wikidata entity IDs through the Wikidata query API, selecting the lowest ID when multiple matches are returned.
  • Entity mapping: The selected Wikidata IDs are mapped to corresponding English Wikipedia URLs through a second query.

Relation classification

The relation-classification setup uses FewRel-derived sentences with marked subject and object mentions, oracle entity IDs, and a relation label to predict. E-BERT is tuned on the development set through a full hyperparameter search, with expected maximum performance tracked against the number of evaluated configurations.

  • The RC dataset is a subset of FewRel containing sentences annotated with subject and object mentions, oracle entity IDs, and relations to predict.
  • Inputs use the BERT wordpiece tokenizer, special markers for subjects and objects, and inserted entity IDs; an example is provided for E-BERT-concat.
  • Hyperparameter tuning varies peak learning rate and number of epochs on the development set, selecting configurations by macro F1 over the same search space as Zhang et al. (2019).
  • Expected maximum relation-classification performance is evaluated as a function of the number of hyperparameter configurations, with E-BERT-concat's best configuration highlighted.
  • Entity linking: For entity linking, AIDA documents are wordpiece-tokenized and recursively split near sentence boundaries when longer than 512 tokens; E-BERT-MLM is tuned using strong-match micro F1.
Loading 1911.03681v2…