PaperlayerView exact revision on arXiv

Source-linked AI summary

COMET: Commonsense Transformers for Automatic Knowledge Graph Construction

Antoine Bosselut, Hannah Rashkin, Maarten Sap, Chaitanya Malaviya, Asli Celikyilmaz, Yejin Choi

arXiv:1906.05317v2cs.CLcs.AI

TL;DR

Commonsense knowledge is loosely structured and often implicit, creating a challenge for automatic knowledge-base construction. COMET adapts pretrained language models on seed tuples to generate novel commonsense knowledge, with human-judged top-1 precision reaching 77.5% for ATOMIC and 91.7% for ConceptNet.

  • Problem

    Commonsense knowledge is often implicit and does not cleanly fit conventional knowledge-base schemas, motivating generative approaches to commonsense completion.

  • Method

    COMET adapts pretrained language-model representations using existing knowledge tuples as seeds to generate novel nodes and edges in commonsense knowledge graphs.

  • Results

    Human evaluators judged up to 77.5% of ATOMIC and 91.7% of ConceptNet generated tuples correct at top 1.

  • Takeaways & Limitations

    The results suggest generative commonsense models could become a plausible alternative to extractive methods for automatic commonsense knowledge-base completion.

  • Takeaways & Limitations

    Some novel generations are simplified forms of training tuples, and the evaluation can be affected by repeated none references in several ATOMIC categories.

Abstract

from arXiv · show

We present the first comprehensive study on automatic knowledge base construction for two prevalent commonsense knowledge graphs: ATOMIC (Sap et al., 2019) and ConceptNet (Speer et al., 2017). Contrary to many conventional KBs that store knowledge with canonical templates, commonsense KBs only store loosely structured open-text descriptions of knowledge. We posit that an important step toward automatic commonsense completion is the development of generative models of commonsense knowledge, and propose COMmonsEnse Transformers (COMET) that learn to generate rich and diverse commonsense descriptions in natural language. Despite the challenges of commonsense modeling, our investigation reveals promising results when implicit knowledge from deep pre-trained language models is transferred to generate explicit knowledge in commonsense knowledge graphs. Empirical results demonstrate that COMET is able to generate novel knowledge that humans rate as high quality, with up to 77.5% (ATOMIC) and 91.7% (ConceptNet) precision at top 1, which approaches human performance for these resources. Our findings suggest that using generative commonsense models for automatic commonsense KB completion could soon be a plausible alternative to extractive methods.

1 Introduction

The paper frames commonsense acquisition as automatic knowledge-base construction, addressing knowledge that is implicit, open-text, and difficult to fit into conventional schemas. COMET uses pretrained language models and existing tuples to generate novel commonsense knowledge for ATOMIC and ConceptNet.

  • Contributions: COMET treats commonsense acquisition as knowledge-base construction and develops a generative approach that produces new nodes and identifies edges between them.
  • Motivation: Commonsense knowledge is difficult to acquire automatically because it often does not fit schemas built around entities and predefined relations.Unlike encyclopedic knowledge, commonsense entities and relations may be natural-language phrases and are often implicit rather than explicitly stated.
  • Motivation: Extractive methods capture only knowledge explicitly mentioned in text, limiting their ability to represent implicit commonsense knowledge.
  • Approach: COMET adapts a pretrained language model using existing knowledge tuples as seeds to generate novel commonsense tuples.The seed tuples provide knowledge for training, while the language model adapts pretrained representations for knowledge generation.
  • Results: 77.5% of generated ATOMIC event tuples and 91.7% of generated ConceptNet relation tuples were judged correct by human evaluators.

2 Learning to Generate Commonsense

COMET adapts pretrained language models to construct commonsense knowledge bases by generating novel knowledge tuples from seed tuples. Its transformer architecture encodes tuple tokens with word and position embeddings, then applies stacked attention, feed-forward, residual, and normalization operations.

  • Learning to Generate Commonsense: COMET adapts language-model representations to add novel nodes and edges to a seed commonsense knowledge graph.The seed tuples provide the knowledge-base structure and relations to be learned.
  • Learning to Generate Commonsense: The task represents knowledge as natural-language tuples {s, r, o} and generates object phrase o from subject s and relation r.For example, (take a nap, Causes, have energy) maps the subject and relation to the object phrase.
  • Transformer Language Model: COMET uses a GPT-style transformer with stacked multi-headed attention and fully connected layers to encode input text.The architecture follows the transformer language model introduced by Radford et al. (2018).
  • Transformer Language Model: Each transformer layer applies multi-headed self-attention, a feed-forward network, layer normalization, and residual connections.The layer transformations use architecturally identical blocks with unique trainable parameters.
  • Transformer Language Model: Input tokens combine word embeddings with position embeddings so the model represents token identity and absolute sequence position.Position embeddings supply ordering information that self-attention does not provide inherently.
  • Transformer Language Model: ATOMIC separates subject, relation, and object with masking tokens, while ConceptNet adds another mask sequence because relations may contain multiple tokens.The configurations use Xs for subjects, Xr for relations, and Xo for objects.

3 Training COMET

COMET is trained to generate object phrases from subject–relation inputs using knowledge tuples from existing commonsense knowledge bases. Training maximizes the conditional likelihood of object tokens, with ATOMIC and ConceptNet serving as seed resources.

  • Training Objective: COMET learns to generate object tokens Xo from the concatenated subject and relation tokens [Xs, Xr].The model is trained to produce the phrase object given the tuple’s phrase subject and relation.
  • Training Objective: Training maximizes the conditional log-likelihood of predicting the object phrase tokens Xo.The token counts of the subject, relation, and object determine the sequence lengths in the objective.
  • Datasets: ATOMIC and ConceptNet provide seed knowledge tuples for training, while COMET is described as domain-agnostic.Other commonsense knowledge resources could also serve as seed sets.
  • Initialization: COMET initializes from final GPT language-model weights and adds special relation tokens for fine-tuning.The added tokens are initialized by sampling from a standard normal distribution.
  • Hyperparameters: The model uses 12 layers, 768-dimensional hidden states, 12 attention heads, 0.1 dropout, GeLU activations, and batch size 64.Dataset-specific hyperparameters are provided separately in the appendix.

4 ATOMIC Experiments

The ATOMIC experiments evaluate COMET’s quality, novelty, decoding behavior, and data efficiency against prior sequence-to-sequence baselines. COMET improves automatic and human-evaluated performance, benefits from pretraining, and produces novel commonsense tuples.

  • Dataset and evaluation: ATOMIC contains 877K tuples spanning nine dimensions of social commonsense about event causes and effects.Events serve as phrase subjects, with relation types representing commonsense dimensions.
  • Overall performance: COMET produces more novel tuple objects than the baselines while improving generation quality.Novelty metrics measure generated tuples and objects absent from the training set.
  • Learning knowledge from language: Pretrained language representations yield a 14% relative improvement in overall human performance over random initialization.The comparison supports transfer from language-model representations to commonsense generation.
  • Effect of decoding algorithm: Greedy decoding leaves a 10% relative performance gap to human evaluation, while beam size 10 produces quality assessments around 55%.Producing more candidates lowers overall performance, and many “none” references contribute to the lower beam-quality score.
  • Efficiency of learning from seed tuples: Using only 1% of training data substantially reduces quality and novelty, whereas training without pretraining performs comparably to using 10% of seed tuples.The experiments vary training-data proportions and initialization to assess data efficiency and the effect of pretraining.

5 ConceptNet Experiments

The ConceptNet experiments evaluate COMET against recurrent baselines using automatic, classifier-based, novelty, and human-quality measures. COMET generates high-quality and novel tuples, while pre-training and relation-language choices are examined as design factors.

  • Setup: COMET is evaluated against bidirectional and forward-only LSTM baselines for ConceptNet tuple generation.The bidirectional baseline learns both sr →o and or →s, while the forward-only LSTM is included separately.
  • Quality: 95.25% classifier-scored accuracy and 91.7% human-rated correctness show that COMET generates high-quality ConceptNet tuples.The classifier score uses a pre-trained Bilinear AVG model, while human evaluation assesses greedily decoded tuples.
  • Novelty: 59.25% of generated tuples are absent from training, and 3.75% of object nodes are novel.These results indicate generation of new edges and some new nodes, although some novel tuples simplify training-set tuples.
  • Novelty: Novelty is measured by normalized minimum edit distance between development-set objects and the nearest training object for the same subject and relation.The metric ranges from zero for identical sequences, excluding stop words, to one for entirely different word sequences.
  • Ablations: Pre-training improves automatic and human evaluations, while natural-language relation representations show limited automatic-metric differences but qualitative generalization examples.Examples include generating “mango IsA fruit” and “dove SymbolOf purity” from limited related training evidence.

6 Related Work

Prior work constructs knowledge bases from expert knowledge or extracted text and develops models for scoring or completing commonsense tuples. COMET instead generates phrases that create new nodes and edges in commonsense graphs using pretrained language models.

  • Knowledge base construction: Earlier knowledge-base construction used expert knowledge, semi-structured extraction, or unstructured text extraction into relational schemas.These approaches target structured knowledge that can be queried for downstream applications.
  • Commonsense generation: Commonsense-generation work on ConceptNet includes tuple-scoring models and joint completion-generation models.Li et al. score full tuples, whereas Saito et al. propose a joint model for completion and generation.
  • Transformers and pre-training: COMET adapts pretrained language models to generate new commonsense graph nodes and edges rather than only extracting or scoring existing tuples.This connects pretrained-language-model adaptation with generative commonsense knowledge-base construction.

7 Conclusion

COMET adapts language-model weights to generate novel and diverse commonsense tuples for automatic knowledge-base construction. Across ATOMIC and ConceptNet, humans frequently judge its generated knowledge correct.

  • Conclusion: COMET adapts language-model weights to produce novel and diverse commonsense knowledge tuples.The framework is introduced for automatic construction of commonsense knowledge bases.
  • Conclusion: On ATOMIC and ConceptNet, COMET frequently generates novel commonsense knowledge that human evaluators deem correct.The conclusion points to extending the approach to other knowledge bases and OpenIE-style tuples from arbitrary seeds.

A Additional Training Details

The additional training details specify separate optimization schedules for ATOMIC, ConceptNet, and the ConceptNet baseline, along with embedding and architecture choices. They also document deviations from prior baseline procedures.

  • Training schedules: ATOMIC COMET training uses a 6.25e-5 maximum learning rate, 100-minibatch warmup, linear decay, 50k minibatches, early stopping, and gradient clipping.The remaining hyperparameters follow Radford et al. (2018).
  • Training schedules: ConceptNet COMET training uses a 1e-5 maximum learning rate, 200-minibatch warmup, linear decay, and 100k minibatches.Other hyperparameters are kept the same as for ATOMIC training.
  • Baseline architecture: The ConceptNet baseline uses 200-dimensional hidden states and embeddings with a single-layer bidirectional encoder and single-layer unidirectional decoder.Relation embeddings are concatenated with decoder word embeddings.
  • Embedding initialization: The baseline uses GloVe rather than fastText embeddings because the reported 200-dimensional setup was incompatible with the available fastText embeddings.The authors cite missing information about the precise corpus used to train the original fastText embeddings.
  • Implementation choices: Baseline deviations include Adam optimization, no weight decay, separate generator training, a forward-only LSTM baseline, and no learning-rate decay.These choices reflect observed validation performance, prior results, or uncertainty about the original schedule.

B.1 Human Evaluations

The human evaluation measures whether generated commonsense tuples are valid, using worker ratings across ATOMIC and ConceptNet settings and multiple generation strategies.

  • Human raters judged generated (s, r, o) tuples for validity across ATOMIC and ConceptNet evaluation sets.ATOMIC used 900 (s, r) pairs from 100 seed concepts; ConceptNet used 1200 pairs.
  • Workers evaluated sets of 2, 5, or 10 sampled completions, with each tuple set rated by 5 workers.These corresponded to Beam-2/5/10 and top-5/10 sampling generations.
  • Greedy sampling produced one completion per (s, r) pair, which workers individually marked valid or invalid.Each greedy-generated tuple was rated by 5 workers.
  • Accuracy is the percentage of distinct worker responses marking a tuple as valid.The measure divides valid responses by 5·|(s,r,o)|.
  • The study also explored multi-task and hierarchical learning configurations over ATOMIC’s commonsense relation taxonomy.The taxonomy groups relations by dimensions such as agent/theme and causes/effects.

D.1 Multi-relation Training

The multi-relation training study compares ATOMIC models trained on individual relations with configurations that share information across related relations, while ConceptNet models jointly use all relations.

  • ATOMIC experiments trained either one model per relation or models with information-sharing configurations across grouped relations.The individual-relation setup is called COMET - 9LM, and the configurations are described in Table 9.
  • ConceptNet models were always trained jointly on all relation types.The authors report no additional ablation experiments because automatic metrics did not differ significantly from COMET - FULL.

D.2 Concept Hierarchy Training

Concept hierarchy training supplies grouped relation information as meta-tokens, enabling shared context across related ATOMIC relations and supporting generation beyond fixed PersonX templates.

  • Concept Hierarchy Training: Hierarchical training appends category meta-tokens to relation tokens so related ATOMIC relations can share information.For example, xReact receives <X>, <POST>, and <Involuntary>, while xWant receives <X> and <POST>.
  • Concept Hierarchy Training: COMET outputs include examples for events such as giving a pep talk, wanting to see a movie, and asking to use someone’s car.These examples illustrate generated ATOMIC commonsense completions for named events.
  • Concept Hierarchy Training: COMET can generalize beyond ATOMIC’s PersonX templates and accept names directly in event inputs.The example involving Eric demonstrates this generalization.
  • Concept Hierarchy Training: The hierarchy organizes relations along dimensions such as participant role, temporal position, and involuntary response.The supplied ATOMIC relation and training-setup tables document these relation groupings and configurations.
  • Concept Hierarchy Training: The evaluation materials include automatic quality and novelty results and human scores comparing regular COMET with category-meta-token COMET.These comparisons are presented in Tables 11 and 12.
Loading 1906.05317v2…