Source-linked AI summary
KG-BART: Knowledge Graph-Augmented BART for Generative Commonsense Reasoning
Ye Liu, Yao Wan, Lifang He, Hao Peng, Philip S. Yu
TL;DR
Generative commonsense reasoning remains challenging because pre-trained language generators often omit relational knowledge and produce implausible sentences. KG-BART augments BART with grounded commonsense graphs and graph attention, outperforming BART on CommonGen and supporting downstream commonsense QA through generated contexts.
Problem
Generative commonsense reasoning is difficult because existing pre-trained language generation models rarely incorporate relational knowledge from commonsense graphs, limiting plausible sentence generation from concept sets.
Method
KG-BART grounds concept-reasoning and concept-expanding graphs and integrates them with BART through knowledge graph-augmented encoder-decoder modules and graph attention.
Results
KG-BART significantly outperforms state-of-the-art pre-trained models on generative commonsense reasoning and produces useful background scenarios for downstream commonsense QA.
Takeaways & Limitations
Capturing concept relations through a knowledge graph enables KG-BART to generate more logical, natural, and high-quality sentences, including for unseen concept sets.
Abstract
from arXiv · showhide
Generative commonsense reasoning which aims to empower machines to generate sentences with the capacity of reasoning over a set of concepts is a critical bottleneck for text generation. Even the state-of-the-art pre-trained language generation models struggle at this task and often produce implausible and anomalous sentences. One reason is that they rarely consider incorporating the knowledge graph which can provide rich relational information among the commonsense concepts. To promote the ability of commonsense reasoning for text generation, we propose a novel knowledge graph augmented pre-trained language generation model KG-BART, which encompasses the complex relations of concepts through the knowledge graph and produces more logical and natural sentences as output. Moreover, KG-BART can leverage the graph attention to aggregate the rich concept semantics that enhances the model generalization on unseen concept sets. Experiments on benchmark CommonGen dataset verify the effectiveness of our proposed approach by comparing with several strong pre-trained language generation models, particularly KG-BART outperforms BART by 5.80, 4.60, in terms of BLEU-3, 4. Moreover, we also show that the generated context by our model can work as background scenarios to benefit downstream commonsense QA tasks.
Introduction
Generative commonsense reasoning remains difficult because text-only generation models often lack the relational knowledge needed to produce coherent scenarios from concept sets. KG-BART addresses this gap by grounding generation in knowledge graphs and integrating that information into BART.
- Generative commonsense reasoning requires generating a natural sentence describing a day-to-day scene from a given concept set.
- Text-only pre-trained generation models capture language information but often ignore knowledge and produce outputs that miss human commonsense.
- Commonsense knowledge graphs provide relational facts that connect concepts and adjunct words, supporting more natural and plausible scenario generation.
- KG-BART grounds concept-reasoning and concept-expanding graphs, then incorporates them into a knowledge graph-augmented BART encoder-decoder.
- The concept-reasoning graph guides relationships among concepts, while the concept-expanding graph models inter-concept and intra-concept relations for more natural outputs.
- KG-BART is reported to outperform state-of-the-art pre-trained models and generate useful background scenarios for downstream commonsense QA.
Problem Formulation
Generative commonsense reasoning maps an unordered set of concepts to a valid, natural-sounding sentence, with a knowledge graph used as auxiliary information. The formulation separates graph construction from final sentence generation.
- A knowledge graph is represented as entities, edges, and relations, with each edge expressed as a subject-relation-object triplet.
- The task maps an unordered set of noun or verb concepts to a valid and natural-sounding language output.
- The graph-construction function takes concepts and knowledge as input to learn a concept-reasoning graph and a hierarchical concept-expanding graph.
- The concept-reasoning graph contains relations between concept pairs, while the concept-expanding graph adds neighboring entities as adjunct information.
Knowledge Graph Grounding
KG-BART grounds concept sets in ConceptNet by constructing reasoning and expanding graphs, selecting relevant neighbors, and learning entity and relation embeddings. These graphs supply structured knowledge for the model's text-generation modules.
- The method constructs and learns embeddings for concept-reasoning and hierarchical concept-expanding graphs from ConceptNet.
- KG-BART hybridizes knowledge-graph and text information in both encoder and decoder modules to address implausible generation from text-only methods.
- Neighboring entities are ranked by summed GloVe cosine similarity to the concept set, and the top-k candidates are added to the expanding graph.
- TransE learns entity and relation embeddings using selected one-hop, two-hop, and three-hop paths between concept pairs.
- For the expanding graph, neighboring-node embeddings and relation embeddings are directly added to the reasoning graph representations.
Graph-Based Encoder-Decoder Modeling
KG-BART extends the BART encoder-decoder with graph-aware modules that integrate textual concepts and commonsense knowledge. The encoder enriches token representations through graph attention, while the decoder uses hierarchical graph attention and combines textual and KG signals for generation.
- Architecture: KG-BART follows BART's encoder-decoder architecture while adding textual and KG-augmented Transformer modules.The encoder processes contextual token information and entity-oriented knowledge, while the decoder generates sentences using commonsense reasoning.
- Encoder: The KG-augmented encoder integrates token embeddings with a concept-reasoning graph to enrich representations using explicit relations.Graph-informed attention is used to learn better intra-concept relations during neural encoding.
- Encoder: Concept subwords are grouped and converted into word-level embeddings with a CNN and max-pooling layer before graph processing.This subword-to-concept integration aligns textual representations with word-level KG concepts.
- Encoder: Graph attention updates concept representations using node and relation features, with multi-head attention aggregating neighboring concepts.Relation representations act as prior constraints, producing richer concept representations that encode relationships among concepts.
- Decoder: The decoder uses hierarchical graph attention to capture intra-concept and inter-concept structure and generate more precise, natural output.It first updates concepts with neighboring nodes, then incorporates intra-concept relations before combining KG and textual attention.
- Decoder: The final decoder representation concatenates KG and textual attention with a residual connection before vocabulary prediction.A softmax layer transforms the resulting representation into the predicted token sequence.
Experiment and Analysis
Experiments on CommonGen evaluate KG-BART against pretrained generation baselines using automatic, human, ablation, qualitative, and downstream-transfer analyses. KG-BART achieves the strongest reported generation results and improves commonsense QA when its generated context is added.
- Dataset and Evaluation: CommonGen contains 77k descriptions over 35k concept sets, with 3–5 concepts per set and unseen test-time concept pairs challenging generalization.The dataset combines crowdsourced and caption corpora and explicitly tests commonsense reasoning in constrained text generation.
- Automatic Evaluation: KG-BART performs best among pretrained models, exceeding T5-large by 7.95% and 8.04% on BLEU-3 and BLEU-4, respectively.It also reports the highest Coverage, 98.68, among baseline pretrained models.
- Human Evaluation: Human evaluation finds KG-BART outputs more reasonable, fluent, and precise than competing models, with significant comparisons against T5-Large and BART.Wilcoxon signed-rank p-values are 1.2e−4 and 2.9e−3 at the 95% confidence level.
- Qualitative Analysis: In a concept-set case study, KG-BART covers all concepts and produces a relatively reasonable, natural, and plausible scenario compared with errors from other models.Other systems omit concepts, produce unusual relationships, repeat content, or contain grammatical errors.
- Attention Analysis: Attention analysis reports that KG-BART assigns more attention to related concept pairs consistent with the knowledge graph than BART does.The paper uses the weight–gym relationship as an example of graph-supported concept association.
- Ablation Study: Ablations show that KG-BART outperforms four variants, supporting the effectiveness of its designed components and KG integration with the pretrained model.The variants remove or alter KG-augmented encoding, entity-position representations, graph attention, or KG-BART pretraining.
- Transfer to Commonsense QA: Adding generated context improves RoBERTa’s CSQA accuracy from 76.22 to KG-BART’s 79.31 and can accelerate training about 2.5 times.At 550 steps KG-BART reaches 75.51, compared with 75.31 for original RoBERTa at 1,400 steps.
Related Work
Prior work incorporates commonsense knowledge into several language-generation applications and integrates entities or knowledge graphs into language understanding. KG-BART extends this direction to constrained generation with a pretrained language model.
- Commonsense in NLG: Existing commonsense-aware generation work spans storytelling, visual storytelling, essay generation, image captioning, evidence generation, and conversational systems.These applications motivate the potential relevance of generative commonsense reasoning to downstream tasks.
- Knowledge-Enhanced Pretraining: Knowledge-enhanced pretrained models such as ERNIE and KEPLER learn joint or aligned representations of words, entities, and textual entity descriptions.The cited approaches primarily address language understanding rather than the constrained generation setting targeted by KG-BART.
Conclusion
The paper presents KG-BART, which uses knowledge-graph relations and neighboring entities to generate high-quality sentences for unseen concept sets. The approach is also described as extensible to other sequence-to-sequence pretrained models.
- Conclusion: KG-BART captures concept relations over a knowledge graph and uses neighboring entities to generate more natural and logical sentences on unseen concept sets.The conclusion characterizes the model as a knowledge-graph-augmented approach built on pretrained BART.
- Conclusion: The approach can be extended to sequence-to-sequence pretrained language generation models such as T5 and MASS.
Broader Impact
KG-BART incorporates knowledge-graph structure and semantics into pre-trained language generation, with potential benefits for commonsense reasoning and extension to other generation tasks and knowledge graphs.
- The authors report that the mechanism improves system performance by a significant margin in experimental evaluations.
- The approach may extend beyond ConceptNet to encyclopedia knowledge graphs such as Freebase and DBpedia.
- The authors suggest adapting KG-BART to dialogue response generation to support conversational AI systems with rational and plausible responses.
Appendix
The appendix describes KG-BART’s implementation settings and identifies recurring generation errors involving verbosity, repetition, commonsense mismatch, and limited global relationship modeling.
- Training Details and Parameters: Training uses 12K selected concepts, 200K sampled five-concept sets, and fixed entity and relation embeddings during pre-training.
- Training Details and Parameters: KG-BART uses six textual Transformer layers, six KG-Augmented Transformer layers, 1,024-dimensional token embeddings, 16 attention heads, and CNN kernel size 2.
- Baseline Implementation: Compared models use beam-search decoding, with beam size 5 for all models.
- Error Analysis: The error analysis identifies three error types in generated sentences with low evaluation scores.
- Error Analysis: KG-BART tends to generate overly long sentences when covering a concept set.
- Error Analysis: KG-BART can repeatedly generate the same concept, producing outputs that differ from the human ground truth.
- Error Analysis: Some generated sentences remain different from human commonsense despite covering the supplied concepts.
- Limitations: The authors characterize KG-BART as learning local relations between concept pairs but not global relationships across phrases in generated sentences.