PaperlayerView exact revision on arXiv

Source-linked AI summary

Entity Cloze By Date: What LMs Know About Unseen Entities

Yasumasa Onoe, Michael J. Q. Zhang, Eunsol Choi, Greg Durrett

arXiv:2205.02832v1cs.CL

TL;DR

Language models are pretrained before many new real-world entities exist, while existing probes do not distinguish entities seen during pretraining from unseen ones across broad entity knowledge. ECBD builds a date-indexed cloze benchmark from Wikipedia and evaluates masked-span perplexity; definitions help models guess spans, but reasoning about completely unseen entities remains challenging.

  • Problem

    Existing entity probes do not test broad real-world entity knowledge while distinguishing entities seen from unseen during pretraining, despite temporal mismatch affecting downstream-task performance.

    Source: §1 · p. 1

  • Method

    ECBD indexes entities by origination date, extracts cloze sentences from their Wikipedia articles, and evaluates language models’ perplexity on masked spans near entity mentions.

    Source: §1 · p. 1, §2.2 · p. 2

  • Results

    Text definitions can meaningfully improve models’ guesses about masked spans, while handling completely unseen entities remains challenging for current language models.

    Source: §1 · p. 2, §6 · p. 5

  • Takeaways & Limitations

    ECBD provides a time-indexed benchmark for studying broad entity inferences and evaluating methods for injecting knowledge into language models.

    Source: §1 · pp. 1–2

    People are notably underrepresented because the gap between birth and prominence makes their origination year difficult to pin down.

    Source: §2.2 · p. 3

Abstract

from arXiv · show

Language models (LMs) are typically trained once on a large-scale corpus and used for years without being updated. However, in a dynamic world, new entities constantly arise. We propose a framework to analyze what LMs can infer about new entities that did not exist when the LMs were pretrained. We derive a dataset of entities indexed by their origination date and paired with their English Wikipedia articles, from which we can find sentences about each entity. We evaluate LMs' perplexity on masked spans within these sentences. We show that models more informed about the entities, such as those with access to a textual definition of them, achieve lower perplexity on this benchmark. Our experimental results demonstrate that making inferences about new entities remains difficult for LMs. Given its wide coverage on entity knowledge and temporal indexing, our dataset can be used to evaluate LMs and techniques designed to modify or extend their knowledge. Our automatic data collection pipeline can be easily used to continually update our benchmark.

1 Introduction

Existing probes do not test broad real-world entity knowledge while distinguishing entities seen from unseen during pretraining. ECBD indexes entities by origination date, builds cloze examples from their Wikipedia articles, and shows that definitions help models guess masked spans.

1 Introduction

  • New entities cannot be in pretrained parametric knowledge, and temporal mismatch affects downstream-task performance.The paper motivates measuring what language models know about real-world entities as new entities continue to arise.Source: §1 · p. 1
  • Existing probes either cover limited knowledge-base relations or add temporal validity without distinguishing new from existing entities, leaving broad entity knowledge under-tested.The paper identifies this as a gap in evaluating knowledge of entities seen versus unseen during pretraining.Source: §1 · p. 1
  • ECBD classifies entities by origination date and builds cloze statements from sentences in their Wikipedia articles.Its masked spans near entity mentions test a wide range of entity-related inferences beyond predefined knowledge-base relations.Source: §1 · p. 1
  • The ECBD dataset contains 35k masked sentences mentioning 2.1K ODIE entities, split by year across 2017–2021, plus 8k masked sentences about popular entities from any time period.Experiments evaluate three pre-trained language models in terms of perplexity.Source: §1 · p. 2
  • Text definitions can meaningfully improve models’ guesses about masked spans, making ECBD useful for benchmarking knowledge-injection methods.The benchmark evaluates masked spans in cloze sentences about entities.Source: §1 · p. 2

2 Entity Cloze by Date

ECBD is designed to test broad entity knowledge and reasoning about entities unseen during pretraining, using origination dates and diverse cloze spans. Its collection pipeline mines dated entities, samples Wikipedia sentences and spans, and produces a dataset with broad vocabulary, though people are difficult to date consistently.

2.1 Task Definition

  • The task pairs an entity and its origination year with a sentence containing its mention and a masked query span, then evaluates perplexity on the masked span.The design aims to assess broad entity knowledge and reasoning about completely unseen entities.Source: §2.1 · p. 2, §2 · p. 2

2.2 Data Collection

  • ODIE mining uses the earliest available Wikidata date among several origination-related date properties as the assumed earliest possible appearance in pretraining data.The date properties include start time, announcement, discovery or invention, inception, and introduction dates.Source: §2.2 · p. 2
  • The pipeline collects sentences from sufficiently long English Wikipedia articles, excluding the first paragraph used as an entity definition and filtering unsuitable mentions and short sentences.Candidate masked spans follow the entity mention, begin within ten words, and do not overlap it; NP spans target relational knowledge, while random spans can cover other knowledge such as actions.Source: §2.2 · pp. 2–3
  • Median span perplexity rises 32.2% when the entity mention is removed, while removing the distance criterion reduces the increase to 25.9%.This analysis indicates that selected spans are related to the entity; the gap test is not used for model-based data filtering.Source: §2.2 · p. 3
  • The entity set spans events, products, and organizations but does not comprehensively capture all entities originating in a given year.People are notably missing because birth and prominence years can differ substantially, making origination year difficult to pin down.Source: §2.2 · p. 3
  • The cloze data has 19K masked-span vocabulary items versus 2K in the compared probe dataset, despite containing fewer entities; the release samples equal numbers of sentences per year.Dev and test sets are split by entity, with no entities shared between them.Source: §2.2 · p. 3, Table 2 · p. 3

3 Experiments

The experiments compare fixed pretrained LMs on cloze spans from seen and unseen entities, testing whether entity mentions and textual definitions affect perplexity. All three models assign higher perplexity to unseen entities, and definitions help more on that subset.

Setup

  • The zero-shot evaluation holds T5-large, BART-large, and GPT-Neo parameters fixed and compares ORIGINAL, NO ENT, RANDOM DEF., and DEFINITION inputs.2020–2021 is grouped as unseen and 2017–2019 as seen; 2020–2021 entities are unseen to T5 and BART, while most are unseen to GPT-Neo, whose training data includes a March 2020 Wikipedia dump.Source: §3 · pp. 3–4

Evaluation Metric

  • The metric averages token-normalized negative log-likelihood first over entities and then examples; GPT-Neo uses left context only, while T5 and BART can also condition on right context.Because model classes differ in context, tokenization, and pretraining tasks, their absolute perplexity values are not directly comparable.Source: §3 · p. 4

Results

  • NO ENT increases perplexity relative to ORIGINAL in every subset, but this does not prove entity knowledge is required because replacing the mention also changes latent stylistic attributes.DEFINITION improves performance over ORIGINAL, while RANDOM DEF. distracts BART and GPT-Neo but slightly improves T5; the authors offer possible explanations for T5’s small gain.Source: §3 · pp. 4–5
  • All three LMs give higher perplexity on unseen entities; for T5, DEFINITION changes perplexity from 19.43 to 13.60 on 2020–2021, versus 13.02 to 11.04 on POPULAR.The authors emphasize relative performance because absolute perplexity across years is sensitive to sentence and entity-type distributions.Source: §3 · pp. 4–5, Table 3 · p. 4
  • For all three LMs, the ORIGINAL-to-DEFINITION performance delta is larger on 2020–2021 than on other subsets.The authors interpret this as definitions being more useful for unseen entities; the POPULAR-set delta is notably smaller than for 2020–2021.Source: §3 · p. 5

4 Use Cases

The dataset is intended for broader knowledge probing as existing benchmarks age, and its time indexing supports evaluating methods for changing or extending model knowledge. It targets broad new-entity inferences rather than only the injection of single facts.

4 Use Cases

  • The authors propose the dataset for general knowledge probing as existing benchmarks become outdated with changing real-world knowledge.They describe its time-indexed entities as suitable for evaluating knowledge-editing approaches and continual knowledge learning.Source: §4 · p. 5
  • The framework is designed to update datasets using the most recent knowledge sources in a controlled manner.Source: §4 · p. 5
  • ECBD supports evaluating knowledge-editing and continual-learning approaches on whether they enable broad new inferences about entities, beyond injecting single facts.Its real-world knowledge coverage can be updated using recent knowledge sources in a controlled manner.Source: §4 · p. 5

5 Related Work

ECBD focuses on entities that did not exist during pretraining and uses origination dates to align them with the real-world timeline. The related work also includes temporal facts, changing word meanings, and cloze tasks drawn from fiction.

5 Related Work

  • Unlike TEMPORALLAMA and SITUATEDQA, which test temporal information about entities in pretraining data, ECBD targets new entities that did not exist during pretraining.ECBD selects entities by origination date, whereas TemporalWiki’s dump differences may not reflect real-world changes during the period.Source: §5 · p. 5
  • Diachronic embeddings model changing word meanings over time, while this work focuses on introducing new concepts rather than rewriting existing ones.The authors note that similar data could be collected for new usages of existing words.Source: §5 · p. 5
  • The Children’s Book Test and LAMBADA use fiction and do not cover real-world entities, despite also using cloze formats.Source: §5 · p. 5

6 Conclusion

The paper presents a time-indexed cloze dataset for studying LMs’ broad inferences about entities. Its analysis finds that handling completely unseen entities remains challenging for current LMs.

6 Conclusion

  • The paper collects 43k cloze-style sentences associated with time-indexed entities.Source: §6 · p. 5
  • The dataset is intended to study language models’ broad inferences about entities across time.Source: §6 · p. 5
  • The analysis finds that handling completely unseen entities remains challenging for current LMs.Source: §6 · p. 5

A Examples of ECBD Sentences

Table 4 provides examples of masked sentences in the ECBD data.

A Examples of ECBD Sentences

B Perplexity per year

Table 5 gives a more fine-grained view of the results in Table 3.

B Perplexity per year

C Perplexity per span type

Table 6 breaks down T5 perplexity by span type: random spans are generally higher perplexity than NP spans, while definitions can help both.

C Perplexity per span type

  • Random spans generally have higher perplexity than NP spans, but adding definitions can help both.Table 6 breaks down T5 perplexity across span types.Source: C Perplexity per span type · p. 7

D Recall@10

Recall@10 is evaluated for T5, with results reported by subset and span type; the authors do not report it for BART or GPT-Neo because extracting predictions is not straightforward for those models.

D Recall@10

  • Recall@10 scores whether a model’s top ten predictions include the gold masked span; T5 predictions are generated with beam size 100 and extracted from the top ten unique sequences.The predicted text spans are extracted between <extra_id_0> and <extra_id_1>.Source: D Recall@10 · p. 7
  • The recall@10 results are reported for each subset, with Table 8 additionally breaking them down by span type.Source: D Recall@10 · p. 7
  • Recall@10 is explored only for T5 because extracting spans for BART is ambiguous when generated right context differs, while GPT-Neo requires conditioning on span length.The authors therefore do not report recall@10 for BART or GPT-Neo.Source: D Recall@10 · p. 7

F Computational Resources

Experiments used an NVIDIA Quadro RTX 8000 and evaluated existing models without fine-tuning; T5 experiments used Hugging Face’s Transformer package.

F Computational Resources

  • All experiments ran on an NVIDIA Quadro RTX 8000; evaluations used existing models without fine-tuning and typically took 15 minutes.Source: F Computational Resources · pp. 7–8
  • Table 4 presents example sentences selected from ECBD’s 2017–2021 subsets.Source: Table 4 · p. 8
  • Table 5 reports test-set perplexity for T5, BART, and GPT-Neo across subsets, while Table 7 reports T5 test-set recall@10 by subset.Source: Table 5 · p. 9, Table 7 · p. 9
  • Tables 6 and 8 report dev-set results for T5 pretrained with 2019 data, giving perplexity and recall@10, respectively, by span type.Source: Table 6 · p. 9, Table 8 · p. 10

Related papers

Loading 2205.02832v1…