PaperlayerView exact revision on arXiv

Source-linked AI summary

CommonGen: A Constrained Text Generation Challenge for Generative Commonsense Reasoning

Bill Yuchen Lin, Wangchunshu Zhou, Ming Shen, Pei Zhou, Chandra Bhagavatula, Yejin Choi, Xiang Ren

arXiv:1911.03705v4cs.CLcs.AIcs.CV

TL;DR

CommonGen addresses the challenge of generating realistic everyday sentences from sets of concepts, a generative commonsense task distinct from choosing among answer options. The paper introduces the task and dataset, evaluates pretrained language models, and reports a substantial gap between the best model and human performance, alongside downstream context-generation benefits.

  • Problem

    Commonsense reasoning benchmarks have largely framed the task as choosing an answer from options, while generating realistically plausible sentences with commonsense remains challenging.

    Source: §1 · p. 1

  • Method

    The paper introduces CommonGen, a constrained generation task and dataset in which models compose everyday-scene sentences from concept sets, collected through controlled crowdsourcing and caption sources.

    Source: §1 · pp. 1–2, §3.1 · p. 3

  • Results

    T5 achieves 28.86% SPICE compared with 52.43% for human performance, and the paper reports that successful CommonGen models can benefit commonsense question answering by generating context.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The findings point to future work on modeling commonsense in language generation, while generated context can benefit downstream tasks such as commonsense-centric question answering.

    Source: §1 · p. 2, §7 · p. 10

    BERTScore overly emphasizes lexical semantics rather than dependencies between words and has low correlation with manual evaluation results.

    Source: §5.2 · p. 7

Abstract

from arXiv · show

Recently, large-scale pre-trained language models have demonstrated impressive performance on several commonsense-reasoning benchmark datasets. However, building machines with commonsense to compose realistically plausible sentences remains challenging. In this paper, we present a constrained text generation task, CommonGen associated with a benchmark dataset, to explicitly test machines for the ability of generative commonsense reasoning. Given a set of common concepts (e.g., {dog, frisbee, catch, throw}); the task is to generate a coherent sentence describing an everyday scenario using these concepts (e.g., "a man throws a frisbee and his dog catches it"). The CommonGen task is challenging because it inherently requires 1) relational reasoning with background commonsense knowledge, and 2) compositional generalization ability to work on unseen concept combinations. Our dataset, constructed through a combination of crowdsourced and existing caption corpora, consists of 79k commonsense descriptions over 35k unique concept-sets. Experiments show that there is a large gap between state-of-the-art text generation models (e.g., T5) and human performance. Furthermore, we demonstrate that the learned generative commonsense reasoning capability can be transferred to improve downstream tasks such as CommonsenseQA by generating additional context.

1 Introduction

CommonGen tests whether machines can generate everyday-scene sentences from supplied concepts, complementing discriminative commonsense tasks. It targets relational reasoning and generalization to unseen concept combinations, and the reported model results remain well below human performance.

1 Introduction

  • Unlike discriminative commonsense tasks that select from choices, CommonGen tests the complementary challenge of generating text with commonsense.Source: §1 · p. 1
  • CommonGen asks a model to generate a sentence describing a day-to-day scene from a given set of concepts.For example, {dog, frisbee, catch, throw} can yield a sentence about a man throwing a frisbee and his dog catching it.Source: §1 · pp. 1–2
  • The task requires relational reasoning over commonsense relations and compositional generalization to unseen combinations of familiar concepts.A grammatical sentence can still be implausible if it violates commonsense relations among the concepts.Source: §1 · p. 2
  • T5 achieves 28.86% SPICE versus 52.43% for humans, while the paper reports that models can generate grammatically sound but implausible sentences.The analysis also reports downstream benefits from generating context for commonsense-centric question answering.Source: §1 · p. 2

2 Task Formulation and Key Challenges

CommonGen maps an unordered set of common objects and actions to a simple, grammatical sentence describing an everyday scenario and using all supplied concepts. Its challenges are selecting plausible commonsense relations and composing concepts that may be unfamiliar in combination.

Task Formulation

  • The input is an unordered set of k common object or action concepts, and the output is a simple, grammatical sentence describing an everyday scenario using all of them.Morphological inflections are allowed; the scenario may depict a static situation or a short series of actions.Source: §2 · p. 2

Relational Reasoning with Commonsense

  • Relational reasoning requires recalling relevant commonsense facts and composing them into a plausible scenario, potentially adding agents or background settings.The underlying chains may draw on spatial, physical, temporal, social, and other knowledge that existing knowledge bases may not record.Source: §2 · pp. 2–3

Compositional Generalization

  • Compositional generalization requires inferring plausible sentences for concept combinations that were unseen in training, including concepts that never co-occurred.The example combines pear, basket, pick, put, and tree even though pear is absent from training and pick never co-occurs with basket.Source: §2 · p. 3

3 Dataset Construction and Analysis

The dataset construction workflow samples concept-sets from caption corpora, collects crowd-written references, and analyzes whether the resulting benchmark tests compositional generalization and commonsense reasoning.

3.1 Collecting Concept-Sets from Captions

  • Concept-sets of three to five concepts are drawn from caption corpora, then scored using shared-sentence coverage, scene diversity, and an inverse-frequency penalty to reduce bias toward frequent concepts.The source corpora include image and video captions; the score guides sampling candidate examples.Source: §3.1 · pp. 3–4

3.2 Crowd-Sourcing References via AMT

  • Evaluation references comprise 10,060 crowd-written sentences for 2.5k concept-sets, with at least three workers assigned to each; low-quality workers and annotations were filtered or removed.Concept-sets whose references mostly made little sense were dynamically replaced.Source: §3.2 · p. 4

3.3 Permutation-Invariant Annotating

  • The mean Spearman rank correlation between input concept order and human reference order is -0.031, indicating no notable ordering influence; 96.97% of references also reorder the concepts.The authors interpret the result as evidence that human annotations are permutation-invariant to the displayed input order.Source: §3.3 · p. 4

3.4 Finalizing Adequate References

  • Inter-annotator agreement and reference diversity both saturate at five references per example, which the authors judge adequate.Agreement is estimated from cosine similarities between SentenceBERT embeddings of pairs of references.Source: §3.4 · pp. 4–5, Figure 4 · p. 5

3.5 Down-Sampling Training Examples

  • Training examples are down-sampled to control overlap with development and test sets, and every pair of concepts in each test set is unseen during training.Development and test examples average four reference sentences each.Source: §3.5 · p. 5

3.6 Analysis of Underlying Common Sense

  • About 60% of five-concept test sets have fewer than one one-hop link among their 10 concept pairs, while nearly half are almost fully connected at two hops, supporting the authors’ assessment of reasonable difficulty.The analysis uses ConceptNet connectivity and also reports distributions of relation types for one- and two-hop connections.Source: §3.6 · p. 5, Figure 5 · p. 5

4 Methods

The methods compare recurrent, transformer, non-autoregressive, and pretrained generation models for CommonGen, using copying, order-robust input handling, and task-specific fine-tuning or decoding choices.

Encoder-Decoder Models

  • Recurrent and Transformer encoder-decoder baselines use attention and copying, while MeanPooling-CopyNet replaces the sequential encoder with mean pooling to eliminate input-order sensitivity.The sequential models are trained and decoded with multiple random concept permutations, whose average performance is reported.Source: §4 · pp. 5–6

Non-autoregressive generation

  • Levenshtein Transformer and ConstLeven represent non-autoregressive baselines that iteratively refine generated sequences, with ConstLeven adding lexical constraints.The authors include these models to test whether iterative refinement may improve performance.Source: §4 · p. 6

Pre-trained Language Generation Models

  • GPT-2, UniLM, UniLM-v2, BERT-Gen, BART, and T5 are fine-tuned on the training data in a sequence-to-sequence format.GPT-2 is conditioned on a concept-set-to-sentence format, while BERT-Gen uses an s2s-ft sequence-to-sequence procedure similar to UniLM’s LM objective.Source: §4 · p. 6
  • T5 receives the prompt “generate a sentence with:” before the concept-set and is decoded with beam search of size 5; dynamic beam allocation does not improve over conventional beam search.The same beam size is used for all compared models.Source: §4 · p. 6

5 Evaluation

The evaluation compares generation metrics and model outputs, tests decoding changes, and studies whether CommonGen-generated contexts help CommonsenseQA. Fine-tuned pretrained models outperform non-pretrained models, while manual analysis and transfer experiments reveal quality differences and potential downstream gains.

5.1 Metrics

  • BLEU, ROUGE, and METEOR measure surface similarity, while concept Coverage measures the average percentage of input concepts present in lemmatized outputs.Captioning metrics CIDEr and SPICE instead focus on associations between mentioned concepts; SPICE uses dependency parses as scene-graph proxies.Source: §5.1 · pp. 6–7

5.2 Experimental Results

  • Fine-tuned pretrained models outperform non-pretrained models across the reported metrics, while CopyNet’s low concept coverage is identified as a key weakness.The table reports test-set results for both model groups; the paper attributes the pretrained models’ advantage partly to relevant pretraining objectives.Source: Table 3 · p. 7, §5.2 · p. 7
  • SPICE and CIDEr rankings most closely track manual coverage-weighted hit@1 rankings, with Spearman’s ρ of 0.94 and 0.91, respectively.METEOR and ROUGE-2 each correlate at 0.88, while BLEU-4 correlates at 0.78.Source: §5.2 · p. 7
  • Manual comparisons find pretrained-model generations more commonsensical, but BART and T5 remain less natural than human references.BART repeats content, while T5 lacks a human agent in the cited case study.Source: §5.2 · p. 8, Figure 6 · p. 8
  • Dynamic beam allocation negatively impacts all tested methods, motivating knowledge-based decoding or re-ranking as future directions.The comparison changes only the decoding method from the Table 3 setup.Source: §5.2 · p. 8, Table 5 · p. 9

5.3 Transferring CommonGen Models

  • Some CommonGen-generated contexts improve CSQA performance, while BERT-gen and ConstLeven cause negative transfer.CG-T5 reaches 74.85% at 550 steps versus 74.77% for original RoBERTa at 1,250 steps, indicating about a 2× training speed-up at those checkpoints.Source: §5.3 · pp. 8–9, Figure 7 · p. 8

6 Related Work

The paper frames CommonGen as a complementary generative commonsense benchmark alongside existing discriminative challenges. It distinguishes the task from constrained generation and commonsense-informed NLG work.

6 Related Work

  • Existing commonsense benchmarks test several capabilities, but success on fine-tuned discriminative tasks does not establish the ability to produce novel assumptions in open-ended generation.The authors present CommonGen as a complementary benchmark for commonsense reasoning in NLG.Source: §6 · p. 9
  • Prior constrained-generation methods target attributes such as sentiment, tense, style, or word order, but the authors say they cannot address CommonGen’s challenges together in one model.Topical story generation targets longer creative stories and is not directly suited to this task.Source: §6 · p. 9
  • Earlier commonsense-informed NLG work spans essays, image captions, video stories, and conversation; CommonGen is presented as a constrained sentence-generation dataset for assessing generative machine commonsense.The authors also point to potential benefits for downstream applications and cite their transferring study as evidence of that potential.Source: §6 · p. 9

7 Conclusion

The paper concludes that CommonGen provides a task and dataset for generative commonsense reasoning, with relational reasoning and compositional generalization as central challenges. Experiments find that leading models remain far from human performance and often produce implausible sentences.

7 Conclusion

  • CommonGen is a constrained text-generation task and large dataset designed to test generative commonsense reasoning.The paper identifies relational reasoning with latent commonsense knowledge and compositional generalization as the task’s inherent challenges.Source: §7 · p. 9
  • Experiments on UniLM, BART, and T5 find a substantial gap from human performance, with outputs that can be grammatical yet realistically implausible.The authors identify modeling commonsense knowledge in language generation as a future research direction.Source: §7 · pp. 9–10
  • The authors hope CommonGen will benefit downstream NLG applications such as conversational systems and storytelling models.Source: §7 · p. 10

A Supplementary Figures and Tables

The supplement provides additional figures and tables referenced in the main paper, including further case studies and complete learning-curve results.

A Supplementary Figures and Tables

  • Supplementary materials include relation-distribution details, four additional case studies, crowd-worker instructions, dev-set results, and the full learning-curve results.The learning-curve table highlights checkpoints and the CG-T5 speed-up discussed in Section 5.3.Source: A Supplementary Figures and Tables · p. 14

B Experimental Details

The appendix documents model implementations, tuning and training settings, and the setup for transferring generated context to CommonsenseQA. It also provides annotation, relation-frequency, and qualitative-analysis materials.

Main experiments

  • Hyperparameters started from framework defaults, with manual tuning over neighboring choices; the authors note further search might improve performance.Learning rates for the last seven models were selected from 1e-3, 1e-4, and 1e-5; batch size was maximized to fit GPU memory.Source: B Experimental Details · pp. 14–15
  • The appendix identifies code locations, parameter counts, and key hyperparameters for baseline methods, and reports their CommonGen dev-set results.The provided material describes Table 6's contents and Table 7's scope, but does not include their cell values.Source: Table 6 · p. 14, Table 7 · p. 15
  • Models used beam search, mostly with beam size 5; T5-large was fine-tuned for about 48 hours on three GPUs.X-CopyNet and LevenTrans training took under 12 hours on one GPU, while the second model group generally took 12–24 hours.Source: B Experimental Details · p. 14

Transferring study experiments

  • CommonsenseQA transfer experiments reused RoBERTa-Large hyperparameters and selected the best of 10 random seeds (42).The setup used batch size 16, learning rate 1e-5, 3,000 updates, 150 warmup steps, dropout 0.1, weight decay 0.01, and Adam epsilon 1e-6.Source: B Experimental Details · p. 14

Appendix materials

  • The annotation interface supplied part-of-speech hints to speed annotation, while retaining sensible sentences with incorrect part of speech.The appendix also shows one/two-hop ConceptNet relation frequencies and four qualitative generation cases, noting that most models perform poorly on most cases.Source: Figure 8 · p. 16, Figure 9 · p. 16, Figure 10 · p. 17

Transferring study experiments

  • CommonsenseQA transfer results are presented on its dev set, with the appendix noting that transfer runs reused the RoBERTa-Large hyperparameters.Table 8 is identified as the transfer-results table; its values are not included in the supplied passage.Source: B Experimental Details · p. 14, Table 8 · p. 18

Related papers

Loading 1911.03705v4…