Source-linked AI summary
Commonsense Knowledge Mining from Pretrained Models
Joshua Feldman, Joe Davison, Alexander M. Rush
TL;DR
Supervised commonsense knowledge mining can underperform on novel data. This paper uses a pretrained bidirectional masked language model to score relational triples, matching supervised performance less closely on ConceptNet but generalizing better to Wikipedia mining.
Problem
Supervised commonsense knowledge-base completion methods are evaluated on held-out data from the same database, leaving their performance on novel sources limited.
Method
The method converts relational triples into sentences and uses a fixed bidirectional masked language model to estimate relation-conditioned pointwise mutual information for validity scoring.
Results
78.8 test-set F1 is comparable to the supervised prototypical approach’s 79.4, while the method achieves superior performance when mining novel commonsense knowledge from Wikipedia.
Takeaways & Limitations
A fixed pretrained model can mine commonsense knowledge from new sources without bias from any one knowledge base’s coverage.
Takeaways & Limitations
The method assumes heads and tails are arbitrary-length word sequences and relations come from a known set of possible relations.
Abstract
from arXiv · showhide
Inferring commonsense knowledge is a key challenge in natural language processing, but due to the sparsity of training data, previous work has shown that supervised methods for commonsense knowledge mining underperform when evaluated on novel data. In this work, we develop a method for generating commonsense knowledge using a large, pre-trained bidirectional language model. By transforming relational triples into masked sentences, we can use this model to rank a triple's validity by the estimated pointwise mutual information between the two entities. Since we do not update the weights of the bidirectional model, our approach is not biased by the coverage of any one commonsense knowledge base. Though this method performs worse on a test set than models explicitly trained on a corresponding training set, it outperforms these methods when mining commonsense knowledge from new sources, suggesting that unsupervised techniques may generalize better than current supervised approaches.
1 Introduction
Commonsense knowledge is difficult to infer because it is rarely stated explicitly, while supervised completion methods trained on ConceptNet generalize poorly to novel data. The paper therefore motivates an unsupervised approach based on pretrained language-model knowledge.
- Commonsense facts are rarely stated explicitly, making them difficult to infer without enormous amounts of data.
- Commonsense knowledge bases provide high-quality information but have low coverage.
- Commonsense knowledge base completion distinguishes valid from fictitious candidate entity-relation-entity triples.
- Existing completion approaches train and test on the same knowledge base, typically using supervised learning on ConceptNet relations.
- Prior evaluations show these supervised methods generalize poorly to novel data, partly because ConceptNet test examples can rephrase training relations.
- The proposed approach avoids training on any particular database and instead relies on a language model’s general world knowledge.
2 Method
The method converts relational triples into candidate sentences, selects a coherent realization, and scores its validity using masked bidirectional language-model estimates of relation-conditioned PMI. It combines conditional and marginal likelihoods, with a weighted score averaged across directions.
- A sentence-generation function maps each relational triple to one sentence, and a scoring model maps that sentence to a validity score.
- Scoring Generated Triples: A masked bidirectional model estimates the conditional likelihood of masked tail words given the remaining sentence context.
- Generating Sentences from Triples: Candidate sentences are created from hand-crafted relation templates and grammatical transformations, then ranked by a pretrained unidirectional model.
- Generating Sentences from Triples: Coherency Ranking selects the highest-likelihood candidate and produces higher-quality sentences than deterministic rules alone.
- Scoring Generated Triples: The final validity score uses weighted PMI from conditional and marginal likelihoods, averaging both head-tail directions to reduce estimate variance.
- Scoring Generated Triples: For multiword tails, the method greedily unmasks the most probable word at each step and multiplies the resulting probabilities.
3 Experiments
The experiments compare unsupervised and supervised approaches for commonsense knowledge completion and Wikipedia mining, while examining how sentence generation affects performance.
- Sentence generation methods: Template + Grammar applies deterministic article, gerund, and pluralization transformations, but can produce incorrect sentences such as “a star in an outer space”.Coherency Ranking differs by selecting combinations of transformations and templates with a language model.
- Experimental setup: The evaluation covers ConceptNet commonsense knowledge base completion and mining commonsense knowledge from Wikipedia.Task 1 uses a balanced test set of 2,400 valid and invalid triples; Task 2 evaluates 3,000 Wikipedia triples and human ratings of the top 100 predictions.
- Task 1: Commonsense Knowledge Base Completion: 78.8 test F1 makes the unsupervised approach comparable to the supervised Prototypical model’s 79.4, but below Factorized’s 89.2 and DNN’s 89.0.These results are reported for the ConceptNet test set.
- Task 2: Mining Wikipedia: 3.00 mean rating out of 4 for the top 100 Wikipedia triples exceeds current supervised methods under Coherency Ranking.The result uses λ = 4 and has a standard error of 0.11; annotator agreement was 0.50 Pearson correlation and 0.23 kappa.
- Analysis: Sentence Generation: Misrepresenting a triple’s meaning harms performance on both tasks, while grammaticality has a high impact on CKBC F1.The analysis motivates templates that more reliably encode the intended relation meaning.
4 Conclusion
The paper presents an unsupervised method for commonsense knowledge base completion using pretrained language-model world knowledge. It performs worse on a held-out ConceptNet test set but generalizes better to Wikipedia mining without prior ConceptNet exposure.
- Method: The method expresses knowledge triples as sentences and estimates weighted point-wise mutual information with a bidirectional masked language model.The estimated score serves as a proxy for triple validity.
- Evaluation: The approach avoids prior exposure to ConceptNet, reducing bias from training on that database when evaluating the held-out test set.The conclusion explicitly contrasts this setting with the method’s lower held-out performance.
- Future work: Future work includes mining non-commonsense facts, generating knowledge beyond candidate databases, and expanding evaluation methods.These directions are presented as possible extensions and ways to strengthen the conclusions.
A Grammatical Transformations
The grammatical transformation procedure modifies triple entities before template injection so generated sentences better match ordinary grammatical forms.
- Transformation pipeline: The procedure applies transformations to the head and tail entities before they are inserted into templates.The implementation uses spaCy for part-of-speech identification and pattern for conjugation and pluralization.
- Transformation rules: Articles are prepended when an entity begins with a noun, adjective, or qualifying verb construction.The article may be indefinite or definite.
- Transformation rules: Infinitive verbs are converted to gerunds, such as “jump” becoming “jumping”.
- Transformation rules: When an entity begins with a number, the following word is pluralized, such as “two leg” becoming “two legs”.
B Hand Crafted Templates
Hand-crafted templates map ConceptNet relations to natural-language sentences, with wording tailored to each relation’s semantic direction and meaning.
- Template design: Each relation is assigned several templates that convert a head-relation-tail triple into candidate sentences.The placeholders {0} and {1} denote the head and tail entities, respectively.
- Relation coverage: Templates cover affirmative, negative, taxonomic, part-whole, usage, capability, location, causal, temporal, motivational, and lexical relations.Examples include IsA, NotIsA, PartOf, UsedFor, CapableOf, Causes, HasPrerequisite, Desires, Synonyms, Antonym, and DefinedAs.
- Relation-specific wording: Some templates explicitly encode relational direction, such as “{1} has {0}” for PartOf and “{0} has {1}” for HasA.
C Most Confident Mistakes
The most confident mistakes include predictions that confuse related concepts, invert properties, or assign implausible relations. These errors appear among the top predictions identified by human annotators as low quality in the Mining Wikipedia task.
- The predictions also contain malformed or mismatched entity-relation pairs, such as a function receiving an element of a and a minister receiving a member of parliament.
- Additional errors include assigning broad category or location relations incorrectly, such as bombing being a war crime and an island being at the other side.
- Several predictions assign implausible relations, such as a violin having a viola or weather being used for heavy rain.
- Other mistakes reverse or conflate conceptual properties, including electric charge being magnetic and a negative number being positive.
- Table 4 lists the top 100 sentence-ranking predictions with λ = 4 that received an average human score below 3 in Mining Wikipedia.
- Table 5 applies the same low-score criterion to the top 100 predictions from the concatenation approach with λ = 7.