Source-linked AI summary
Language Models as Knowledge Bases: On Entity Representations, Storage Capacity, and Paraphrased Queries
Benjamin Heinzerling, Kentaro Inui
TL;DR
Existing LM-as-KB work handled only a restricted set of single-token entities, while the capacity for storing many entity-linked facts and querying them through paraphrases remained limited. The paper compares three entity representations, measures storage capacity, and studies paraphrased queries, finding support for million-entity storage and both zero-shot and few-shot query transfer. It concludes that current LM architectures can serve as knowledge bases when extended with an entity-representation component.
Problem
Prior LM-as-KB work was limited to 21k single-token entities, leaving large-scale entity representation and paraphrased querying underexplored.
Method
The paper compares symbolic, surface-form, and continuous entity representations, measures fact-storage capacity, and evaluates transfer from memorized statements to paraphrased queries.
Results
Current LM architectures can store millions of entity-linked facts and support zero-shot transfer for highly similar queries plus few-shot transfer for less similar queries after 5 to 100 instances.
Takeaways & Limitations
The findings provide a proof-of-concept that language models can serve as knowledge bases when extended with components for representing entities.
Takeaways & Limitations
The experiments do not address more complex structured-KB queries such as 1-to-n relations, multihop inference, numerical ranges, or time- and location-qualified facts.
Abstract
from arXiv · showhide
Pretrained language models have been suggested as a possible alternative or complement to structured knowledge bases. However, this emerging LM-as-KB paradigm has so far only been considered in a very limited setting, which only allows handling 21k entities whose single-token name is found in common LM vocabularies. Furthermore, the main benefit of this paradigm, namely querying the KB using a variety of natural language paraphrases, is underexplored so far. Here, we formulate two basic requirements for treating LMs as KBs: (i) the ability to store a large number facts involving a large number of entities and (ii) the ability to query stored facts. We explore three entity representations that allow LMs to represent millions of entities and present a detailed case study on paraphrased querying of world knowledge in LMs, thereby providing a proof-of-concept that language models can indeed serve as knowledge bases.
1 Introduction
The paper frames LM-as-KB systems around storing many facts about many entities and retrieving them through natural-language queries. It compares entity representations, estimates storage capacity, and studies transfer from memorized statements to paraphrased queries.
- Motivation: LMs can memorize factual statements and recover some stored facts through suitable natural-language queries.This motivates treating language models as alternatives or complements to structured knowledge bases.
- Entity coverage: 21k entities with single-token vocabulary names define the earlier setting, excluding entities such as United Kingdom and Sesame Street.The limitation arises because typical LM vocabularies contain at most about 100k entries, while knowledge bases contain millions of entities.
- Entity representations: Three representations scale LM-as-KB systems to millions of entities: symbolic vocabulary entries, surface-form subword sequences, and continuous embeddings.Surface forms use a sequence decoder, whereas continuous representations encode each entity as an embedding.
- Representation trade-offs: Symbolic representation is most accurate but computationally expensive and dependent on entity-linked training data; surface representation is efficient but less accurate for long names.Continuous representation also requires entity-linked data while being more computationally efficient than symbolic representation.
- Storage capacity: 125 million-parameter Transformers memorize 1 million Wikidata triples at 95 percent accuracy or 5 million triples at 79 percent accuracy.These measurements provide empirical storage-capacity evidence for LM architectures.
- Paraphrased querying: Zero-shot transfer occurs for highly similar paraphrased queries, while 5 to 100 finetuning instances enable successful few-shot transfer for less similar queries.The study treats soft natural-language querying as a central distinction from symbolic queries in SQL or SPARQL.
- Contributions: The paper formulates LM-as-KB requirements as storing many facts involving many entities and querying stored facts.Its contributions compare entity representations, establish empirical capacity bounds, and study transfer to paraphrased queries.
2 World Knowledge in Language Models
The paper situates language models’ world knowledge between text-based learning and structured knowledge bases. It emphasizes that rare entities and unstated concepts are difficult to acquire from text alone, motivating representations that make entities directly predictable.
- Limitations of text-based learning: Text-only language models lack grounding in perception and experience, limiting their access to meanings not explicitly stated in text.The supplied discussion contrasts explicit textual facts with knowledge readily available in structured knowledge bases.
- Rare entities: Rare entities are difficult for language models to learn from text alone because they are rarely mentioned.This creates a long-tail knowledge problem for language-model training.
- Research landscape: Knowledge-intensive models span a spectrum from purely text-based language models to representations of structured knowledge bases.Text-generation and retrieval models have been successful in knowledge-intensive tasks, while entity-reifying models make entities directly predictable.
3 Entity Representations
The paper evaluates symbolic, surface-form, and continuous entity representations for storing and retrieving synthetic world-knowledge statements involving up to millions of entities. Symbolic representation is most accurate but computationally expensive, while surface-form and continuous representations scale to 6 million entities with different accuracy and data requirements.
- Experimental setup: Synthetic data pairs Wikidata relations with manually authored templates to measure how accurately language models memorize and retrieve facts.The experiments use up to 5 million statements involving 1 million entities and 10 million statements involving 6 million entities.
- 3.1 Symbolic Representation: Symbolic representation adds every entity to the vocabulary and predicts the masked object with a vocabulary softmax.This approach achieves very high accuracy, but scaling the softmax to 6 million entities is impractical.
- 3.1 Symbolic Representation: 97 percent of 5 million statements are memorized correctly by randomly initialized RoBERTa-base with symbolic representation.The result uses a vocabulary containing 1 million entities.
- 3.2 Surface Form Representation: Surface-form representation generates each entity’s canonical name as a sequence, enabling coverage of all 6 million Wikipedia entities.Evaluation requires an exact match of the complete generated entity name, and the best randomly initialized Transformer memorizes 10 million statements with 90 percent accuracy.
- 3.2 Surface Form Representation: Surface-form errors concentrate on infrequent entities with long names, especially mentions occurring only one to ten times and spanning at least 12 subwords.The error analysis distinguishes correctly memorized statements from incorrect ones and annotates selected clusters by object entity.
- 3.3 Continuous Representation: Continuous representation predicts an embedding and retrieves the nearest pretrained entity embedding, scaling to 6 million entities without relying on entity names.RoBERTa without pretraining memorizes 67 percent of 10 million statements, but the authors describe these results as lower bounds because continuous prediction remains early-stage.
- Representation trade-offs: Symbolic representation is most accurate but computationally expensive; surface form is efficient without entity-linked data, while continuous representation is more efficient than symbolic but requires entity-linked data.Surface-form accuracy is especially affected by longer entity names, whereas continuous representation avoids that name-length difficulty but remains sensitive to entity frequency.
4 LM Capacity for Storing Facts
LM storage capacity increases with model size, but memorization accuracy declines as the number of facts grows. Continuous entity representations provide the best storage efficiency, although entity count and parameter-count differences constrain comparisons.
- Memorization accuracy declines as the number of facts increases, while larger models can store more facts.A 384-layer-size Transformer reaches 65% accuracy when memorizing 3 million facts.
- At 80% accuracy, capacity scales approximately linearly: 55M-parameter models store 1 million facts, while 160M-parameter models store 7 million.
- Increasing the entity set from 1 million to 6 million reduces continuous-representation accuracy from 85% to 79%, while surface representation drops by 2%.
- The representation comparison does not control for differences in trainable parameter counts across model architectures.The authors introduce storage efficiency to make comparisons across models with differing parameter counts fairer.
- Continuous representation is the most storage-efficient representation and is therefore used in the remainder of the work.
5 Querying Stored Facts
The paper tests whether LMs can retrieve memorized facts from paraphrased queries rather than only reproducing memorized wording. Transfer is strongest for similar queries, while dissimilar variants require few-shot finetuning and may still fail.
- The experiment tests whether memorized facts transfer to paraphrased queries rather than reflecting generic association.Control statements assign the same subject different predicates and objects, requiring the model to use the predicate when answering.
- Small query changes preserve high recall, but more divergent variants sharply reduce zero-shot transfer.Ellipses yield 95% accuracy versus 98% for original statements, while other variants range from 3% to 13%.
- Zero-shot transfer accuracy is strongly negatively correlated with the representational distance between original and variant query templates (Pearson −0.68).
- Few-shot finetuning enables transfer to less similar queries with 5 to 100 instances, whereas the least similar variants may require up to 500 instances.
- Without pretraining, transfer to dissimilar variants remains poor, reaching only 1% to 4% accuracy even after finetuning on 500 instances.The results are presented as evidence that pretraining enables handling paraphrased queries beyond rote memorization and generic association.
6 Limitations and Conclusions
The paper acknowledges limits in its evidence and scope: experiments use Wikidata, focus on entity representations, and address only relation-triple storage and retrieval. It nevertheless concludes that current language models can serve as knowledge bases when extended with an entity-representation component, while noting further benefits from paraphrased queries and controlled fact memorization.
- Limitations: Experiments use only Wikidata, so results may differ for knowledge bases with other graph structures.The authors note that some graph types may be easier for language models to memorize than others.
- Limitations: The analysis focuses on entity representations and does not examine how to represent relation predicates or entire relation triples.The authors also distinguish their adaptive-pretraining setup from simply training a language model to memorize statements.
- Limitations: Structured knowledge bases support queries beyond the relation-triple queries studied here, including 1-to-n relations, multihop inference, numerical ranges, and time- or location-qualified facts.The triple-storage and retrieval experiments are presented as a proof of concept rather than coverage of these more complex query types.
- Conclusions: The paper concludes that current language-model architectures meet the requirements for representing many entities, storing many facts, and answering varied queries when extended with an entity component.The conclusion specifically includes the ability to handle paraphrased queries.
- Conclusions: Fact memorization combined with paraphrase fine-tuning allows precise control over which facts a language model learns.The authors present this as an additional potential benefit of the LM-as-KB paradigm.
A Overview: world knowledge in natural language processing
Table 2 surveys approaches for using world knowledge in natural language processing, spanning purely text-based methods, mixed text-and-structured-KB methods, and structured-KB methods.
- Approach spectrum: Table 2 organizes world-knowledge approaches from purely text-based methods through mixed approaches to methods operating on structured knowledge bases.The table presents this progression as a range of approaches in natural language processing.
C Random sample of English statements generated from Wikidata relations
The section provides a random sample of English statements generated from Wikidata relations, covering temporal, familial, geographic, organizational, artistic, and sports-related facts.
- Temporal relations: The sample includes temporal relations, such as one event or season following another.Examples include European Rowing Championships and Sheffield Wednesday F.C. seasons.
- Geographic relations: The sample includes geographic and administrative relations, such as locations, named places, bodies of water, and territorial containment.Examples include Shibam, Makabana, Reynoldsburg, Anlauter, and Châu Thành.
- Cultural relations: The sample includes media and cultural relations, including performers, screenwriters, cast members, directors, and production companies.Examples span films, songs, television episodes, and media organizations.
- Organizational and classification relations: The sample includes organizational and classification relations, such as ownership, parent taxa, industrial connections, and sports or competition affiliations.Examples include Gamecock Media Group, Dione juno, Phoenix Air, and Xavier Musketeers.
- Family relations: The sample includes family relations, including spouses, parents, siblings, and given names.Examples mention Krishna Vamsi, Malcolm Forbes, Edward Harley, Ferenc Keser˝u, and Sidnei.
E Embeddings of Wikidata entities
The paper constructs Wikidata entity embeddings by encoding entity features and concatenating their representations, while comparing alternative training signals and hierarchical organization. These embeddings support the broader goal of representing millions of entities in language-model knowledge bases.
- Entity embedding construction: The method includes feature-specific autoencoders for training embeddings of Wikidata entities.The figure identifies the embeddings as being trained with feature-specific autoencoders.
- Entity embedding construction: Entity embeddings are built by encoding collected entity features and concatenating the resulting dense feature representations.Textual features use RoBERTa-base with sequence-to-sequence autoencoders, while quantities use fixed-sized representations based on common quantity types.
- Training alternatives: Using original pretrained entity embeddings with Euclidean loss converges more slowly and achieves lower memorization accuracy than using spherical entity embeddings as targets.This comparison concerns alternative prediction targets for the entity representations.
- Hierarchical representation: A hierarchy over all Wikidata entities is created with a KD-tree and used as training signal for handling large entity inventories.The approach uses each entity’s position in the hierarchy as supervision.
F.2 Training entity embeddings with negative sampling
The experiments replace fixed pretrained entity embeddings with randomly initialized embeddings updated during training using in-batch negative samples, but this performs worse in memorization and convergence.
- Randomly initialized entity embeddings were updated during training with 1–50 in-batch negative samples.This follows a standard method in knowledge-base embedding and entity-retrieval work.
- Compared with fixed pretrained entity embeddings without negative sampling, the approach achieved lower memorization accuracies and slower convergence.
F.3 Updating pretrained entity embeddings during training
Updating entity embeddings during training increased computational costs without improving memorization accuracy; the accompanying graph-type experiment compares memorization across uniform and scale-free graphs.
- Updating entity embeddings during training increased trainable parameters, memory usage, and training time without producing higher memorization accuracies.
- The graph experiment compares uniform Erdos-Renyi and scale-free Barabasi graphs by increasing relation triples until models reach 99 percent memorization accuracy.Graph edges represent relation triples, and models predict the relation object from subject and predicate.