PaperlayerView exact revision on arXiv

Source-linked AI summary

ExplaGraphs: An Explanation Graph Generation Task for Structured Commonsense Reasoning

Swarnadeep Saha, Prateek Yadav, Lisa Bauer, Mohit Bansal

arXiv:2104.07644v3cs.CLcs.AI

TL;DR

ExplaGraphs addresses the limits of discriminative commonsense reasoning tasks by requiring models to predict stance and generate structured explanations. It introduces a graph-based task, dataset, collection and evaluation framework, but baseline models still show a large gap from human performance.

  • Problem

    Discriminative commonsense-reasoning tasks inadequately evaluate whether models can use and explain relevant commonsense knowledge rather than exploit shortcuts or biases.

  • Method

    ExplaGraphs combines stance prediction with generation of connected, directed acyclic commonsense explanation graphs, supported by Create-Verify-And-Refine collection and multi-level evaluation.

  • Results

    79% of graphs contain external commonsense nodes, while baseline models have difficulty generating meaningful graph explanations and remain far below human performance.

  • Takeaways & Limitations

    ExplaGraphs provides a structured benchmark for studying generative commonsense reasoning and motivates future work on higher-quality graph-based explanations.

  • Takeaways & Limitations

    Graph collection is challenging because annotators must create connected, acyclic graphs and verify their semantic consistency and stance inference.

Abstract

from arXiv · show

Recent commonsense-reasoning tasks are typically discriminative in nature, where a model answers a multiple-choice question for a certain context. Discriminative tasks are limiting because they fail to adequately evaluate the model's ability to reason and explain predictions with underlying commonsense knowledge. They also allow such models to use reasoning shortcuts and not be "right for the right reasons". In this work, we present ExplaGraphs, a new generative and structured commonsense-reasoning task (and an associated dataset) of explanation graph generation for stance prediction. Specifically, given a belief and an argument, a model has to predict if the argument supports or counters the belief and also generate a commonsense-augmented graph that serves as non-trivial, complete, and unambiguous explanation for the predicted stance. We collect explanation graphs through a novel Create-Verify-And-Refine graph collection framework that improves the graph quality (up to 90%) via multiple rounds of verification and refinement. A significant 79% of our graphs contain external commonsense nodes with diverse structures and reasoning depths. Next, we propose a multi-level evaluation framework, consisting of automatic metrics and human evaluation, that check for the structural and semantic correctness of the generated graphs and their degree of match with ground-truth graphs. Finally, we present several structured, commonsense-augmented, and text generation models as strong starting points for this explanation graph generation task, and observe that there is a large gap with human performance, thereby encouraging future work for this new challenging task. ExplaGraphs will be publicly available at https://explagraphs.github.io.

1 Introduction

ExplaGraphs addresses the limits of discriminative commonsense reasoning by requiring models to predict stance and generate explicit commonsense explanation graphs. Its dataset, collection framework, evaluation suite, and baseline results establish a challenging benchmark with a substantial human–model performance gap.

  • Motivation: Discriminative commonsense tasks can permit shortcuts and fail to evaluate whether predictions are supported by relevant reasoning.The paper therefore emphasizes generative commonsense reasoning that composes and reveals the knowledge needed for a prediction.
  • Task: ExplaGraphs requires predicting whether an argument supports or counters a belief and generating a commonsense graph that explains that stance.The graphs represent reasoning over social, cultural, or taxonomic commonsense knowledge.
  • Dataset: Create-Verify-And-Refine collection produces non-trivial, complete, and unambiguous explanations, with graph quality improving up to 90%.79% of graphs contain external commonsense nodes, while graphs must connect concepts and unambiguously infer the stance.
  • Evaluation and Models: The paper introduces multi-level automatic and human evaluation plus a structured model that jointly predicts nodes and edges under global graph constraints.The evaluation examines stance and graph consistency, local edge correctness, global stance revelation, and graph matching.
  • Results: 20% model accuracy versus 84% human performance shows a large gap in generating reasoning graphs for this task.The proposed structured, BART, and T5-based models remain relatively weak at producing meaningful graph explanations.

2 Related Work

Prior work includes extractive, natural-language, and structured explanations, alongside discriminative and emerging generative commonsense tasks. ExplaGraphs differs by generating commonsense-augmented explanation graphs specifically for stance prediction.

  • Structured Explanations: Existing NLP explanation datasets span extractive rationales, free-form explanations, and structured explanations such as graphs and fact chains.The related work situates ExplaGraphs within structured explanation research while distinguishing its graph-generation focus.
  • Research Gap: Unlike prior commonsense and stance-prediction studies, ExplaGraphs generates explicit commonsense-augmented graph explanations for stance prediction.The authors describe this as the first work to explore this combination for stance prediction.

3 EXPLAGRAPHS Task Definition

ExplaGraphs defines stance prediction as jointly inferring support or counter and generating a connected commonsense explanation DAG. The task constrains graph structure and semantics so explanations explicitly and unambiguously connect arguments to beliefs.

  • Task Definition: The task jointly predicts whether an argument supports or counters a belief and generates its corresponding commonsense explanation graph.Graphs use short concept phrases connected by predefined commonsense relations, with internal and external concepts filling knowledge gaps.
  • Graph Representation: Explanation graphs are connected directed acyclic graphs whose concepts are internal or external and whose relations are adapted from ConceptNet.The relation inventory removes generic relations, merges similar relations, and adds negated counterparts.
  • Data Collection: The data collection process first gathers belief–argument–stance triples and then collects corresponding explanation graphs through Create-Verify-And-Refine.The framework uses two stages, with verification focused on semantic correctness and stance inference.
  • Structural Correctness: Structural constraints limit concept length, require 3–8 edges, include at least two belief and two argument concepts, and enforce a connected DAG.These constraints support quality control, explicit reasoning chains, and simpler evaluation.
  • Semantic Correctness: Semantic correctness requires coherent facts and explanations that are non-trivial, complete, and unambiguous.Graphs must use the argument rather than merely paraphrasing the belief and must support reliable stance inference.

4 Dataset Collection

EXPLAGRAPHS data were collected in two crowdsourced stages, followed by iterative verification and refinement of commonsense explanation graphs. The process targets connected, structurally valid graphs whose reasoning is complete and unambiguous.

  • Collection stages: The dataset is collected in two stages: annotators first create belief–argument–stance triples, then construct corresponding commonsense explanation graphs.Stage 1 covers beliefs and supporting or counter arguments; Stage 2 collects a graph for each triple.
  • Graph creation: Annotators construct graphs as 3–8 facts forming a connected DAG with concepts from both the belief and argument.The interface checks structural constraints and prompts annotators to ensure graphs are non-trivial, complete, and unambiguous.
  • Verification and refinement: Verification asks annotators to infer the stance from the belief and graph, while refinement edits incorrect graphs by adding, removing, or replacing facts.Refinement is performed by a qualified annotator who did not create the initial graph.
  • Quality control: Graph quality rises from 67% to 81% and then 90% semantically correct graphs across three rounds of the Create-Verify-And-Refine process.The framework iterates between verification and refinement to improve semantic correctness.

5 Dataset Analysis

EXPLAGRAPHS contains 3,166 samples with graph structures intended to capture substantial commonsense reasoning. Most graphs include external commonsense nodes, and many are non-linear or comparatively deep.

  • Graph characteristics: 79% of graphs contain external commonsense nodes, while 58% are non-linear and average graph depth is 4.These statistics indicate that many samples require background commonsense knowledge and complex reasoning structures.
  • Collection challenges: Graph collection is challenging because annotators must create connected, acyclic graphs and verify their semantic consistency.This difficulty motivates the quality-control process used during annotation.

6 Evaluation Metrics

The evaluation framework checks explanation graphs at multiple levels, from stance and structural validity to semantic correctness, edge importance, and graph similarity. It combines automatic model-based metrics with graph-matching measures and human evaluation.

  • Multi-level evaluation: The framework evaluates stance accuracy first, then structural correctness, before assessing semantic correctness and graph matching.Only samples passing earlier levels proceed to later graph-quality evaluations.
  • Semantic correctness: Semantic Correctness Accuracy uses a RoBERTa classifier to judge whether graph edges are coherent and whether the graph implies the original stance.The metric predicts incorrect, support, or counter from a belief–graph pair.
  • Graph matching: G-BERTScore matches predicted and gold graph edges using BERTScore-style edge similarities, while Graph Edit Distance counts graph edits needed for isomorphism.GED is normalized so exact matches score 0 and larger distances indicate poorer matches.
  • Local explanation quality: Edge Importance Accuracy removes one edge at a time and marks it important when a stance model’s confidence decreases.The metric reports the macro-average of important edges in predicted graphs.

7 Models

The paper compares reasoning and rationalizing model families, using a commonsense-augmented structured model alongside BART and T5 graph generators. The structured model predicts nodes and edges while enforcing graph constraints during inference.

  • Model families: The experiments compare reasoning models that generate graphs before stance prediction with rationalizing models that generate graphs after stance prediction.Both families use fine-tuned RoBERTa for stance prediction.
  • Evaluation: Table 3 reports model performance across the EXPLAGRAPHS test-set metrics, including human semantic-correctness evaluation.The comparison includes the structured RE-SP model and RA-BART/T5 and RE-BART/T5 variants.
  • Text-generation models: BART and T5 generate explanation graphs as linearized strings, with graph edges ordered by depth-first traversal during training.These models provide text-generation baselines for graph generation.
  • Structured model: The commonsense-augmented structured model predicts internal nodes, incorporates external commonsense, predicts edges, and enforces connectivity and acyclicity.Its inference procedure uses an Integer Linear Program for graph constraints.

8 Experiments and Analysis

RE-SP outperforms BART/T5 across automatic and human evaluation, but generated explanations remain far below human quality. Analyses attribute the gap to missing structural constraints, weak commonsense reasoning, and low semantic correctness.

  • SP vs BART/T5: RE-SP is the best-performing model across all automatic metrics and human evaluation, with structural constraints improving StCA and consequently SeCA.Its advantage also appears in G-BS, GED, and EA.
  • Explanation Impact (RA vs RE): Conditioning stance prediction on generated graphs reduces confidence, indicating that the models often fail to produce helpful explanations.RE models predict stance from their generated graph, unlike RA models, which predict stance first without it.
  • Metrics’ Upper Bound: Models achieve sufficiently high stance accuracy but only 10-20% graph semantic correctness, while ground-truth graphs raise SA from 87% to 91% and reach 83% SeCA.This reveals a gap exceeding 60% between model and human performance.
  • Human-Metric Correlation for SeCA: Human verification matches the SeCA model metric 68% of the time, although human evaluation remains necessary because semantic correctness is challenging.RE-SP is the best model under this comparison.
  • SP vs BART/T5: BART and T5 fail to generate many structurally correct graphs because they lack explicit constraints, whereas RE-SP generates richer nonlinear graphs with more nodes, edges, and external nodes.The reported graph statistics attribute these differences to RE-SP’s individual components.
  • Analysis of Generated Graphs: RE-SP’s generated graphs can include external commonsense nodes, but randomly selected outputs also contain incorrect relations and incoherent facts.Figure 6 contrasts one correct graph with one incorrect graph.

9 Discussion and Future Work

The discussion presents explanation graphs as a broadly applicable representation for structured reasoning beyond stance prediction. Their explicit reasoning structures can be adapted to other NLU tasks while preserving task-specific relations and constraints.

  • Applicability: Explanation graphs may extend to NLI, sentiment classification, and question answering by representing context concepts, external concepts, semantic relations, and DAG-like reasoning constraints.Stance prediction serves as a motivating sentence-pair inference use case, with support and counter corresponding to entailment and contradiction.
  • Why Graphs: Graphs capture complex dependencies without redundancy and make task-specific constraints such as connectivity and acyclicity easier to impose than natural-language explanations.These properties motivate their use as structured explanations for reasoning tasks.

10 Conclusion

The paper introduces EXPLAGRAPHS as a benchmarked generative task for commonsense explanation graph generation in stance prediction. Its evaluation and initial model show that producing high-quality explanations remains difficult, motivating better graph-based methods.

  • Conclusion: EXPLAGRAPHS combines a generative, structured commonsense-reasoning task with a benchmarking dataset for explanation graph generation in stance prediction.
  • Conclusion: Automatic metrics and an initial structured model demonstrate the difficulty of generating high-quality commonsense-augmented graphical explanations and motivate future graph-based research.

Ethical Considerations

The paper combines crowdsourced data collection, verification, and refinement with graph-based modeling and ablations that expose the task’s scope and difficulty. Its ethical boundary is that the dataset mainly reflects English-speaking US annotators’ views on debate topics.

  • Ethical Considerations: The dataset mostly reflects English-speaking US annotators’ views, although both support and counter arguments were collected for completeness.
  • Ethical Considerations: 198 workers from the US and Australia were selected using a 96% HIT approval threshold and at least 1000 approved HITs.
  • Data Collection: Data collection used pre-HAMLET plus three HAMLET rounds, with dev and test topics excluded from pre-HAMLET to preserve example difficulty.
  • Quality Control: Annotators underwent onboarding, stance-label verification, and graph-quality checks, while personal feedback improved incoherent, trivial, or incomplete graphs.
  • Modeling: The graph model predicts nodes and 29 edge classes, then uses an ILP to enforce connected-DAG structure during inference.
  • Difficulty Analysis: Higher reasoning depth and non-linear graph structure reduce graph-generation performance, with drops in graph metrics at depth >= 6 and semantic correctness for non-linear structures.

E.4 Quantitative Analysis of Generated Explanation Graphs from RE-T5

The paper illustrates Reasoning-T5 outputs through examples pairing beliefs, arguments, stances, and generated commonsense explanation graphs. These examples show that the model typically produces linear chains with fewer external commonsense nodes than RE-SP.

  • Generated Graphs: Reasoning-T5 typically generates linear chains with far fewer external commonsense nodes than RE-SP.
  • Examples: The qualitative examples pair each belief and argument with a stance and a corresponding commonsense explanation graph.
  • Examples: The examples include both support and counter stances across topics such as celibacy, entrapment, organ donation, autonomous cars, marriage, and plastic surgery.
  • Examples: One example labels the argument that marriage is society’s backbone as counter to the belief that marriage does not mean much.
Loading 2104.07644v3…