PaperlayerView exact revision on arXiv

Source-linked AI summary

KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representation

Xiaozhi Wang, Tianyu Gao, Zhaocheng Zhu, Zhengyan Zhang, Zhiyuan Liu, Juanzi Li, Jian Tang

arXiv:1911.06136v3cs.CL

TL;DR

PLMs struggle to capture factual knowledge, while conventional knowledge embedding methods underuse textual information. KEPLER jointly optimizes knowledge embedding and masked language modeling objectives, improving NLP performance and supporting inductive knowledge embedding; its evaluation also highlights limitations in probing reliability and cross-setting comparisons.

  • Problem

    PLMs generally cannot capture world facts well, while knowledge embedding methods do not fully exploit textual entity information.

  • Method

    KEPLER jointly optimizes knowledge embedding and masked language modeling objectives while encoding entity descriptions into entity embeddings.

  • Results

    KEPLER brings improvements across a wide range of NLP tasks and shows effectiveness as a knowledge embedding model, especially in the inductive setting.

  • Takeaways & Limitations

    Wikidata5M provides a large-scale KG benchmark with aligned entity descriptions and transductive and inductive data splits for future research.

  • Takeaways & Limitations

    LAMA-style probing remains vulnerable to superficial clues and vocabulary effects, motivating better knowledge probing frameworks.

Abstract

from arXiv · show

Pre-trained language representation models (PLMs) cannot well capture factual knowledge from text. In contrast, knowledge embedding (KE) methods can effectively represent the relational facts in knowledge graphs (KGs) with informative entity embeddings, but conventional KE models cannot take full advantage of the abundant textual information. In this paper, we propose a unified model for Knowledge Embedding and Pre-trained LanguagE Representation (KEPLER), which can not only better integrate factual knowledge into PLMs but also produce effective text-enhanced KE with the strong PLMs. In KEPLER, we encode textual entity descriptions with a PLM as their embeddings, and then jointly optimize the KE and language modeling objectives. Experimental results show that KEPLER achieves state-of-the-art performances on various NLP tasks, and also works remarkably well as an inductive KE model on KG link prediction. Furthermore, for pre-training and evaluating KEPLER, we construct Wikidata5M, a large-scale KG dataset with aligned entity descriptions, and benchmark state-of-the-art KE methods on it. It shall serve as a new KE benchmark and facilitate the research on large KG, inductive KE, and KG with text. The source code can be obtained from https://github.com/THU-KEG/KEPLER.

1 Introduction

PLMs capture linguistic patterns but not factual knowledge well, while conventional KE methods do not fully use textual entity information. KEPLER unifies both through shared text encoding and joint KE–MLM training, alongside the Wikidata5M benchmark.

  • PLMs learn language from unstructured corpora but generally fail to capture sparse, complex world facts.
  • KGs provide structural facts, while textual entity descriptions offer information that can strengthen knowledge representation.
  • KEPLER encodes entity descriptions and entities into one semantic space using a shared PLM, jointly optimizing KE and MLM objectives.
  • KEPLER integrates factual knowledge into language representations, retains PLM language understanding, and can be used without additional inference overhead.
  • KEPLER uses textual descriptions to produce embeddings for unseen entities, supporting inductive knowledge embedding beyond conventional transductive methods.
  • Wikidata5M contains about 5M entities, 20M triplets, aligned Wikipedia descriptions, and transductive and inductive data splits.

2 KEPLER

KEPLER unifies knowledge embedding and masked language modeling in a shared Transformer-based text encoder. It represents entities from descriptions, supports several knowledge-embedding variants, and trains the objectives jointly.

  • Architecture and training: KEPLER jointly trains knowledge embedding and masked language modeling objectives while sharing a Transformer-based text encoder.The model uses the same encoder for both objectives rather than adding external entity-linking or knowledge-integration layers.
  • Architecture and training: KEPLER avoids additional inference overhead compared with vanilla PLMs because it adds training objectives without modifying the Transformer encoder or requiring entity linking.This design is intended to make downstream application as easy as using RoBERTa.
  • Knowledge embedding: Entity descriptions are encoded as entity embeddings for knowledge-graph training, with variants that also encode relation descriptions or condition entity embeddings on relations.The paper explores three approaches: entity descriptions as embeddings, entity and relation descriptions as embeddings, and relation-conditioned entity embeddings.
  • Training objectives: The knowledge-embedding objective uses negative sampling and a TransE-style scoring function, while masked language modeling predicts randomly selected input tokens.The reported MLM procedure selects 15% of positions and applies masking, random replacement, or no change according to fixed proportions.
  • KEPLER variants: KEPLER-Wiki is the principal model, using Wikidata5M and entity descriptions as embeddings, while KEPLER-WordNet, KEPLER-W+W, KEPLER-Rel, and KEPLER-Cond provide alternative configurations.KEPLER-Rel performs worse when relation descriptions are short and homogeneous, whereas KEPLER-Cond achieves superior link-prediction results in both transductive and inductive settings.

3 Wikidata5M

Wikidata5M is a large-scale KG dataset constructed with aligned Wikipedia descriptions and transductive and inductive splits to support KEPLER and future research. Its scale and coverage make link prediction more challenging, while standard evaluation constrains validation and test-set size.

  • Dataset construction and settings: Wikidata5M combines large-scale knowledge facts, aligned entity descriptions, and transductive and inductive data splits.The dataset is built from Wikidata and English Wikipedia to address limitations of existing KG benchmarks.
  • Dataset construction and settings: The dataset collection aligns Wikidata entities with Wikipedia pages and discards entities lacking pages or descriptions shorter than five words.The final KG retains relational facts whose entities are retained and whose relations have nonempty Wikidata pages.
  • Dataset construction and settings: Wikidata5M contains 4,594,485 entities, 822 relations, and 20,624,575 triplets, covering various domains and exceeding the scale of other KG datasets.The dataset statistics are presented alongside four widely used benchmarks.
  • Dataset construction and settings: In the inductive setting, entities and triplets are disjoint across splits, requiring models to embed unseen entities from descriptions or other side features.This setting is more challenging and relevant to KGs whose entities grow over time.
  • Limitations: Standard link-prediction evaluation limits Wikidata5M validation and test sets to thousands of triplets because each episode requires |E|×|T|×2 KE score calculations.The paper identifies more efficient evaluation as future work for large-scale KE.
  • Benchmark results: Popular KE methods perform significantly worse on Wikidata5M than on datasets such as FB15K-237 and WN18RR, indicating greater challenge from its scale and coverage.The benchmark uses GraphVite because the original implementations do not scale to Wikidata5M; evaluation ranks entities for head and tail prediction using standard metrics.

4 Experiments

The experiments evaluate KEPLER across NLP and knowledge-embedding tasks, comparing it with language-model and knowledge-enhanced baselines. KEPLER generally performs strongly, while results also expose limitations related to model size, training resources, and evaluation settings.

  • Relation Classification: KEPLER-Wiki significantly outperforms all baselines on TACRED, making it the principal model for subsequent experiments.KEPLER-WordNet provides only marginal improvement over Our RoBERTa, while KEPLER-W+W underperforms KEPLER-Wiki.
  • NLP Tasks: KEPLER-Wiki achieves the best performance among BASE-size PLMs in most FewRel settings.ERNIE and KnowBert can degrade in several settings, whereas KEPLER improves on FewRel 2.0 and generalizes across domains.
  • FewRel: KEPLER still outperforms knowledge-enhanced baselines on FewRel despite not using entity linking or entity-embedding pre-training.Those mechanisms provide ERNIE and KnowBert special advantages on entity-span tasks.
  • Knowledge Embedding: KEPLER underperforms TransE under its main training setup but surpasses TransE† when both use one negative sample.KEPLER uses negative sampling size 1 and 30 epochs, compared with typical KE settings of 64 or more negatives and 1000 epochs.
  • Inductive Setting: KEPLER outperforms DKRL and RoBERTa by a large margin in the inductive Wikidata5M setting, although its results remain below practical application requirements.The inductive setting tests producing embeddings for entities from textual descriptions, and variant comparisons remain consistent with the transductive setting.
  • Evaluation Caveat: Inductive and transductive KE scores should not be directly compared because the inductive evaluation ranks entities from a much smaller entity set.The evaluations involve 7,475 versus 4,594,485 entities, respectively.

5 Analysis

The analyses examine KEPLER’s joint objectives, factual knowledge probing, efficiency, and behavior across entity frequencies. Results indicate benefits for factual knowledge integration, rare-entity modeling, and knowledge extraction, with probing limitations remaining.

  • Joint Objective: Jointly optimizing KE and MLM objectives is important: both single-objective ablations significantly underperform full KEPLER on TACRED.The comparison includes an MLM-only model, Our RoBERTa, and a KE-only model, KEPLER-KE.
  • Knowledge Probing: KEPLER consistently outperforms the vanilla PLM baseline on almost all LAMA settings except ConceptNet.The exception is attributed to ConceptNet’s focus on commonsense rather than factual knowledge.
  • Knowledge Probing: RoBERTa-based models perform worse than vanilla BERTBASE on LAMA, highlighting vulnerability to vocabulary-sensitive probing.The authors associate this pattern with different vocabularies and call for better knowledge probing frameworks.
  • Efficiency: KEPLER introduces no additional parameters or inference computations compared with vanilla PLMs and avoids entity linking or entity-embedding fusion.This design is presented as beneficial for time-sensitive applications.
  • Entity Frequency: KEPLER’s improvements over RoBERTa are generally larger at low entity-masking rates, while KEPLER still outperforms RoBERTa when all entity mentions are masked.The authors connect the residual advantage to the KE objective’s role in learning fact-related text.
  • Entity Frequency: The least-frequent “0%-20%” entity setting yields marginally higher improvement, but KEPLER significantly improves modeling for both rare and frequent entities.Table 13 averages F-1 scores over five runs and defines settings by entity frequency in Wikipedia.
  • Knowledge Extraction and Storage: KEPLER-Wiki improves over Our RoBERTa in both masked-entity and only-entity TACRED settings, supporting stronger extraction and storage of knowledge.The KE-only model performs poorly in general language understanding while retaining some knowledge in entity names.

6 Related Work

Related work spans language-model pre-training, knowledge-enhanced PLMs, knowledge embedding, and inductive graph embedding. KEPLER differs by unifying text and knowledge objectives without adding entity-linking or knowledge-integration mechanisms at inference.

  • Pre-training in NLP: NLP pre-training evolved from distributed word representations and recurrent auto-encoders to Transformer-based models such as GPT, BERT, XLNet, and RoBERTa.Later work also explored additional tasks and larger parameterizations for PLMs.
  • Knowledge-Enhanced PLMs: Knowledge-enhanced PLMs incorporate factual information through matching objectives, linked entity embeddings, integrated entity linkers, relation-based training, or entity replacement.Examples include MTB, ERNIE, KnowBert, and related generation and entity-focused methods.
  • Knowledge-Enhanced PLMs: Other approaches align entity embeddings with word embeddings or inject knowledge through neural adapters for continuous learning.E-BERT and K-Adapter represent these alternative injection strategies.
  • Knowledge Embedding: Conventional knowledge embedding models score relational triplets using functions based on translations, matrix multiplication, or complex operations.TransE, DistMult, ComplEx, and RotatE exemplify these scoring-function approaches.
  • Inductive Embedding: Textual metadata can support inductive knowledge embedding, while graph-neighborhood methods require unseen nodes to be surrounded by known nodes.The paper leaves fully inductive KE without additional features as future work.

7 Conclusion and Future Work

The paper concludes that KEPLER unifies knowledge embedding and pre-trained language representation through joint KE and MLM training, while Wikidata5M supports research on large knowledge graphs. Future work targets smoother semantic unification and better knowledge probing.

  • Conclusion: KEPLER jointly trains KE and MLM objectives to align factual knowledge and language representation in a shared semantic space.The paper reports effectiveness across NLP and knowledge-embedding applications.
  • Conclusion: Wikidata5M is proposed as a large-scale knowledge-graph dataset to facilitate future research.Its role is stated alongside KEPLER’s unified modeling contribution.
  • Future Work: Future work will explore smoother semantic-space unification, alternative KE forms and training objectives, and improved knowledge-probing methods.The probing work is intended to clarify knowledge-integration mechanisms.
Loading 1911.06136v3…