PaperlayerView exact revision on arXiv

Source-linked AI summary

COMET-ATOMIC 2020: On Symbolic and Neural Commonsense Knowledge Graphs

Jena D. Hwang, Chandra Bhagavatula, Ronan Le Bras, Jeff Da, Keisuke Sakaguchi, Antoine Bosselut, Yejin Choi

arXiv:2010.05953v2cs.CL

TL;DR

Pretrained language models express only limited varieties of commonsense knowledge, motivating knowledge resources that provide complementary information. The paper introduces ATOMIC20, evaluates it against existing commonsense graphs, and trains COMET models on it; COMET(BART) shows a 47.9% absolute advantage over GPT2-XL on ATOMIC20.

  • Problem

    Language models struggle to express large varieties of implicit commonsense knowledge, while commonsense knowledge resources differ in coverage and quality.

  • Method

    The paper constructs ATOMIC20 and evaluates commonsense graphs by the complementary information they provide to pretrained language models, including COMET adaptation experiments.

  • Results

    47.9% absolute difference separates COMET(BART) and GPT2-XL on ATOMIC20, while ATOMIC20 contains high-accuracy knowledge across novel relations not found in existing resources or expressible by language models.

  • Takeaways & Limitations

    Commonsense resources should prioritize knowledge that pretrained language models are less likely to know or express, such as HinderedBy relations.

  • Takeaways & Limitations

    The experiments cannot include Cyc because its research license was discontinued and its commercial license cost over $1M.

Abstract

from arXiv · show

Recent years have brought about a renewed interest in commonsense representation and reasoning in the field of natural language understanding. The development of new commonsense knowledge graphs (CSKG) has been central to these advances as their diverse facts can be used and referenced by machine learning models for tackling new and challenging tasks. At the same time, there remain questions about the quality and coverage of these resources due to the massive scale required to comprehensively encompass general commonsense knowledge. In this work, we posit that manually constructed CSKGs will never achieve the coverage necessary to be applicable in all situations encountered by NLP agents. Therefore, we propose a new evaluation framework for testing the utility of KGs based on how effectively implicit knowledge representations can be learned from them. With this new goal, we propose ATOMIC 2020, a new CSKG of general-purpose commonsense knowledge containing knowledge that is not readily available in pretrained language models. We evaluate its properties in comparison with other leading CSKGs, performing the first large-scale pairwise study of commonsense knowledge resources. Next, we show that ATOMIC 2020 is better suited for training knowledge models that can generate accurate, representative knowledge for new, unseen entities and events. Finally, through human evaluation, we show that the few-shot performance of GPT-3 (175B parameters), while impressive, remains ~12 absolute points lower than a BART-based knowledge model trained on ATOMIC 2020 despite using over 430x fewer parameters.

1 Introduction

The paper questions whether pretrained language models express commonsense knowledge broadly and proposes evaluating knowledge graphs by the complementary information they provide. It introduces ATOMIC20 and shows advantages for symbolic accuracy and neural knowledge-model transfer.

  • Motivation: Pretrained language models perform best on narrowly specific, language-like or ontological commonsense assertions, leaving their directly accessible knowledge limited.The paper emphasizes that implicit commonsense facts involving complex relationships remain difficult for neural models to express.
  • Evaluation goal: The paper proposes evaluating commonsense knowledge resources by the complementary information they bring to pretrained language models.This reframes knowledge graphs as resources for learning implicit knowledge representations, not only as static graphs for retrieval.
  • Symbolic comparison: ATOMIC20 covers more correct facts across more diverse commonsense types than any existing publicly available commonsense knowledge resource.The comparison also finds substantial exclusivity between knowledge graphs, underscoring the difficulty of covering general commonsense knowledge at scale.
  • Neural transfer: ATOMIC20 transfer produces COMET models with the largest increase over their seed language model across the commonsense types it covers.The result supports constructing resources with examples of knowledge not readily found in language models.
  • Neural transfer: COMET-ATOMIC20 beats GPT-3 despite using 400x fewer parameters.The comparison is used to demonstrate the utility of high-quality symbolic knowledge for commonsense generalization.

2 Background

The background contrasts major commonsense knowledge graphs and explains why language models alone may not express implied, relational commonsense knowledge. COMET adapts pretrained models using graph tuples to generate novel commonsense knowledge.

  • Commonsense knowledge graphs: CONCEPTNET contains 3.4M English tuples across 36 relations focused mainly on taxonomic, lexical, and physical commonsense knowledge.Its crowdsourced and database-derived knowledge is described as fairly accurate, though quality varies by source.
  • Commonsense knowledge graphs: ATOMIC contains 880K tuples across 9 relations covering social commonsense, event dynamics, conditional statements, causes, effects, and mental states.The dataset was collected and validated completely through crowdsourcing.
  • Commonsense knowledge graphs: TRANSOMCS contains 18.48M automatically extracted tuples but has significantly lower precision and performs poorly as an adaptation resource.Its relations were mapped from syntactic parses of web sources using CONCEPTNET relation types.
  • Commonsense knowledge graphs: The comparison selects CONCEPTNET, ATOMIC, and TRANSOMCS because they retain closed, comparable relational types.Other resources may provide wider varieties of fine-grained relations.
  • Language models as knowledge bases: Prior language-model evaluations often use narrow, text-explicit taxonomic subsets, while many commonsense facts are implied and relationally complex.This limits conclusions about whether language models broadly encode commonsense knowledge.
  • Language models as knowledge bases: COMET takes a head phrase and relation as input and generates a target phrase after training on knowledge-graph tuples.This adaptation enables novel and valid commonsense knowledge tuples from CONCEPTNET and ATOMIC.

3 ATOMIC20

ATOMIC20 is constructed as a large commonsense repository focused on social, physical, and event-centered everyday inference. Its design selectively retains high-quality, language-model-complementary knowledge and adds novel relation types.

  • Resource scope: ATOMIC20 contains 1.33M everyday inferential knowledge tuples about entities and events.The resource covers social and physical aspects of common human everyday experiences.
  • Relation inventory: ATOMIC20 introduces 23 relations across social-interaction, physical-entity, and event-centered commonsense categories.The three categories contain 9, 7, and 7 relation types respectively.
  • Novel relations: ObjectUse captures everyday objects, including necessary, expected, atypical, and feasible creative uses.The construction collected over 130K object-use pairs, such as using a popcorn bucket to hold popcorn or as a hat.
  • Novel relations: HinderedBy represents defeasible events by collecting hindrances to goals that can inform counterfactual reasoning.The paper collected over 100K hindrance tuples for existing ATOMIC20 events.
  • Data integration: The resource selectively integrates 172K tuples from the 3.4M English tuples in CONCEPTNET v5.7.The selected subset was incorporated into ATOMIC20 rather than copying the full English graph.
  • Data curation: Construction removes low-weight, dictionary-based, lexical, and co-occurrence relations that language models can readily recover.The remaining data is identified through elimination, crowdsourcing, and post-processing to emphasize qualitative human experiences.

4 Symbolic Knowledge Graph Comparison

The study compares ATOMIC20 with prominent commonsense knowledge graphs using crowdsourced accuracy and pairwise coverage measures. ATOMIC20 achieves the strongest reported accuracy and widest coverage, while its integrated relations show both improvements and limitations.

  • Accuracy: ATOMIC20 outperforms other knowledge graphs in crowdsourced accuracy, while ATOMIC ties with CONCEPTNET and TRANSOMCS performs substantially worse.Accuracy was assessed through human judgments of randomly sampled tuples.
  • Accuracy: ATOMIC20 improves accuracy over CONCEPTNET for some aligned relations, while most cognates are significantly higher or similarly accurate.The comparison attributes some gains to preprocessing during CONCEPTNET integration.
  • Accuracy: ATOMIC20 performs worse than CONCEPTNET on ObjectUse/UsedFor and MadeUpOf/MadeOf, partly because atypical affordances can be judged farfetched.Its MadeUpOf union of MadeOf, PartOf, and some HasA relations did not improve quality.
  • Coverage: Coverage precision and recall are computed pairwise after mapping relations and tuples across knowledge graphs.Coverage precision measures source tuples found in the target, whereas recall measures target tuples recovered by the source.
  • Coverage: ATOMIC20 shows the widest coverage, recalling all of ATOMIC and just under half of CONCEPTNET.ATOMIC and CONCEPTNET overlap little because they emphasize social and physical commonsense respectively; TRANSOMCS overlaps little with all three.

5 Neural Knowledge Graph Comparison

The paper tests whether pretrained language models can transfer knowledge from commonsense graphs to unseen entities, concepts, and events. Knowledge-graph adaptation substantially improves generation, and COMET(BART) outperforms much larger zero-shot language models in human evaluation.

  • Models: COMET models are trained by fine-tuning pretrained language models to generate tuple tails from heads and relations, with GPT2-XL also evaluated zero-shot.The study uses GPT2-XL and BART as adapted models and converts relations into English prompts for comparison.
  • Evaluation setup: Knowledge-graph training enables language models to generalize relations to unseen entities, concepts, or events under a non-overlapping-head evaluation split.The split forces models to apply learned relationships to entities not seen during knowledge-graph fine-tuning.
  • Results: Knowledge-graph adaptation produces a large performance gap over zero-shot GPT2-XL across ATOMIC20, ATOMIC, and CONCEPTNET.The result indicates that language-only pretraining does not let models precisely express this knowledge directly.
  • Results: ∼12 points separate GPT-3 and COMET(BART) in human evaluation, despite GPT-3 having 175B parameters versus COMET(BART)’s 406M.The paper also reports a large gap across automated metrics.
  • Results: COMET models trained on ATOMIC20 and ATOMIC outperform those trained on CONCEPTNET, while TRANSOMCS-trained models fail to generalize effectively to new entities.The results support the value of knowledge that is difficult to learn from pretraining and of accurate training examples.

6 Discussion

The discussion argues that pretrained language models do not reliably express all commonsense knowledge in zero-shot settings, motivating carefully designed knowledge resources. It emphasizes difficult knowledge categories, broad relation coverage, and high factual accuracy while acknowledging uncertainty about whether models encode knowledge internally.

  • Pretrained knowledge: 47.9% absolute difference separates COMET(BART) and GPT2-XL on ATOMIC20, although adversarial evaluation leaves open whether pretrained models encode knowledge they cannot express.The discussion distinguishes knowledge stored in parameters from the learned interface used to verbalize it.
  • Resource design: CSKG designers should prioritize knowledge categories that pretrained language models are less likely to know or express directly.For HinderedBy, human plausibility rose from 1.3% for zero-shot GPT2-XL to 85% for COMET(BART).
  • Resource design: Commonsense resources should combine broad relationship coverage with carefully validated facts, because large but low-quality TRANSOMCS training data did not support unseen-entity generalization.The discussion presents accuracy and coverage as joint design goals.

7 Conclusion

The paper formalizes commonsense knowledge graphs as transfer-learning resources and introduces ATOMIC20, designed to contain knowledge that pretrained language models struggle to express. ATOMIC20 provides high-accuracy novel relations and supports training knowledge models that generate high-quality tuples on demand.

  • ATOMIC20 is designed as a transfer-learning resource containing relations that are challenging for pretrained language models to express.
  • ATOMIC20 contains high-accuracy tuples across novel relations absent from existing commonsense knowledge graphs or not expressible by language models.
  • ATOMIC20 can train language models to generate high-quality commonsense tuples on demand.

A ATOMIC 2020 Details

ATOMIC20 organizes commonsense knowledge into physical-entity, social-interaction, and event-centered categories, using relations that describe objects, people, events, causes, goals, and temporal sequences. Its population combines inherited, integrated, and crowdsourced tuples, including defeasible goal hindrances and object affordances.

  • ATOMIC20 relations fall into three categories: physical-entity, social-interaction, and event-centered commonsense.
  • Physical-Entity Commonsense: Physical-entity relations describe object affordances, composition, properties, locations, and capabilities relevant to interacting with the world.
  • Social-Interaction Commonsense: Social-interaction relations represent socially triggered mental states, intentions, reactions, needs, desires, effects, and perceived attributes.
  • Event-Centered Commonsense: Event-centered relations capture causal forces, hindrances, explanations, temporal order, and plausible fillers for event scripts.
  • Tuple Population: The resource incorporates 877K existing ATOMIC social-interaction tuples and crowdsources additional tuples, including over 100K event hindrances and 33K isFilledBy tuples.
  • CONCEPTNET Integration: CONCEPTNET integration required filtering factual, lexical, underspecified, and automatically extracted relationships that did not reliably reflect qualitative commonsense experience.

Human Evaluation

The paper uses human-readable relation templates for crowdsourced human evaluations, translating telegraphic knowledge-graph labels into language forms suitable for prompts.

  • Human-readable language forms were created for relation labels and displayed in crowdsourced evaluations.

KB Accuracy & Coverage

The accuracy and coverage analysis examines judgment patterns across knowledge graphs and shows that ATOMIC20 emphasizes context-dependent, defeasible commonsense knowledge. TRANSOMCS performs better on its top subsets but still trails other knowledge bases.

  • No-judgment tuples commonly involve specialized concepts, highly specific entities, or worker-hedged candidates, especially in TRANSOMCS.
  • TRANSOMCS’s top 1% and top 10% subsets have higher accuracy than its complete-set values but still lag behind other knowledge bases.
  • Across knowledge bases, social-interaction tuples are usually accepted as sometimes or likely true, while physical-entity tuples more often receive always or often ratings.
  • ATOMIC20’s social-interaction and event-centered categories contain proportionally more human-crowdsourced knowledge than its physical-entity category.
  • ATOMIC20 prioritizes qualitative commonsense over factual information, making much of its knowledge defeasible and context-dependent.

C Neural Knowledge Graph Details

This section describes dataset splits, tuple-scale expansion, and neural knowledge-model training and evaluation procedures for ATOMIC20 and other commonsense knowledge graphs. It also reports that pretraining improves commonsense-task generalization.

  • Dataset and splits: ATOMIC20 preserves the train, development, and test assignments of tuples inherited from ATOMIC and CONCEPTNET.
  • Dataset and splits: ATOMIC20 provides over 50% more tuples than the initial ATOMIC version.
  • Neural knowledge models: GPT2-XL is fine-tuned on formatted knowledge-graph tuples, using the head and relation to generate the tail after a special generation token.The tuple format places [GEN] before the tail and [SEP] after it.
  • Neural knowledge models: BART is fine-tuned as a denoising sequence-to-sequence model using pretrained representations and tuned training hyperparameters.The described search varies batch size, learning rate, and random seed.
  • Evaluation: GPT-3 evaluation uses relation-specific templates with five randomly selected training examples for priming and three random seeds.
  • Evaluation: 54.9% Accept, 44.9% Reject, and 0.18% No Judgement were obtained by the un-pretrained BART control, substantially below COMET(BART).The authors interpret this drop as evidence that pretraining provides generalization needed for commonsense tasks.

D Additional Reproducibility Items

This section supplies reproducibility details for automated evaluation, computing infrastructure, data and code release, and supplementary collection and visualization materials. It also identifies how several figures encode commonsense categories.

  • Evaluation reproducibility: Automated metrics are reported on full test sets and confirm the findings from Section 5 and the human-evaluation subsets.
  • Experimental setup: Experiments used eight NVIDIA Quadro RTX 8000 GPUs with 48 GB of GDDR6 memory each and fixed random seeds where possible.Model and hyperparameter details are provided in Appendix C.
Loading 2010.05953v2…