PaperlayerView exact revision on arXiv

Source-linked AI summary

Abductive Commonsense Reasoning

Chandra Bhagavatula, Ronan Le Bras, Chaitanya Malaviya, Keisuke Sakaguchi, Ari Holtzman, Hannah Rashkin, Doug Downey, Scott Wen-tau Yih, Yejin Choi

arXiv:1908.05739v2cs.CL

TL;DR

Abductive reasoning is important for interpreting narratives, but has rarely been studied in natural language. The paper introduces αNLI, αNLG, and the ART dataset to evaluate language-based abduction. Models remain substantially below humans on the tasks, while ART transfer helps mainly when target datasets are small.

  • Problem

    Abductive reasoning is central to narrative understanding, yet natural-language abductive reasoning has rarely been studied because prior work largely focused on rigid formal logic.

  • Method

    The paper introduces αNLI and αNLG, creates the ART dataset of 20,000 narratives and over 200,000 explanatory hypotheses, and evaluates neural language models.

  • Results

    68.9% αNLI accuracy is achieved by the best BERT model versus 91.4% for humans, while generative models remain below human performance on αNLG.

  • Takeaways & Limitations

    The analysis identifies reasoning types that deep pre-trained language models fail to perform despite strong entailment-NLI performance, and ART pre-training helps related low-data tasks.

  • Takeaways & Limitations

    ART pre-training does not provide significant improvement for target datasets with large amounts of training data.

Abstract

from arXiv · show

Abductive reasoning is inference to the most plausible explanation. For example, if Jenny finds her house in a mess when she returns from work, and remembers that she left a window open, she can hypothesize that a thief broke into her house and caused the mess, as the most plausible explanation. While abduction has long been considered to be at the core of how people interpret and read between the lines in natural language (Hobbs et al., 1988), there has been relatively little research in support of abductive natural language inference and generation. We present the first study that investigates the viability of language-based abductive reasoning. We introduce a challenge dataset, ART, that consists of over 20k commonsense narrative contexts and 200k explanations. Based on this dataset, we conceptualize two new tasks -- (i) Abductive NLI: a multiple-choice question answering task for choosing the more likely explanation, and (ii) Abductive NLG: a conditional generation task for explaining given observations in natural language. On Abductive NLI, the best model achieves 68.9% accuracy, well below human performance of 91.4%. On Abductive NLG, the current best language generators struggle even more, as they lack reasoning capabilities that are trivial for humans. Our analysis leads to new insights into the types of reasoning that deep pre-trained language models fail to perform--despite their strong performance on the related but more narrowly defined task of entailment NLI--pointing to interesting avenues for future research.

1 INTRODUCTION

Abductive reasoning infers the most plausible explanation for incomplete observations and is central to narrative understanding, yet has rarely been studied in natural-language NLP. The paper introduces language-based abductive reasoning tasks and the ART dataset to investigate this capability.

  • Abductive reasoning infers the most plausible explanation for incomplete observations and supports narrative understanding and reading between the lines.
  • Formal-logic approaches have dominated prior abductive-reasoning research but are too rigid to generalize to natural-language complexity.
  • The paper presents the first study of language-based abductive reasoning and probes deep neural models using natural language representations.
  • The paper introduces αNLI, a multiple-choice task that selects the more likely explanation from a pair of hypotheses given a narrative context.
  • The ART dataset contains 20K narratives and over 200K explanatory hypotheses, supporting evaluation of the proposed tasks.
  • 68.9% accuracy is achieved by the best BERT αNLI baseline, below the 91.4% human performance, while GPT2-based αNLG remains below humans.

2 TASK DEFINITION

The paper defines abductive reasoning tasks over pairs of temporally ordered observations and hypotheses. αNLI selects the more plausible explanation, while αNLG generates a plausible explanation in natural language.

  • Abductive Natural Language Inference: αNLI represents each instance as two observations and two hypothesis choices, then selects the more plausible explanation.
  • Abductive Natural Language Inference: O1 is the observation at time t1, while O2 is the later observation at time t2 > t1.
  • Abductive Natural Language Inference: The plausible hypothesis h+ explains both observations, whereas h− is implausible or less plausible for them.
  • Abductive Natural Language Generation: αNLG generates a valid hypothesis h+ given O1 and O2 by maximizing P(h+|O1, O2).

3 MODELS FOR ABDUCTIVE COMMONSENSE REASONING

The paper develops probabilistic and neural model families for selecting explanations from observations and generating explanations conditioned on narrative inputs. These models vary in how they use observations and commonsense information.

  • Probabilistic Framework for αNLI: The αNLI objective selects the hypothesis h* that is most probable given all available observations and their commonsense implications.
  • Probabilistic Framework for αNLI: The probabilistic framework compares independence assumptions from a hypothesis-only baseline to models using one or both observations.
  • Probabilistic Framework for αNLI: The Linear Chain model uses both observations but treats their influences independently, imposing conditional independence between O1 and O2 given H.
  • Probabilistic Framework for αNLI: The Fully Connected model jointly models O1, H, and O2 and can in principle combine information across both observations.
  • Abductive Natural Language Generation: The αNLG architecture integrates nine COMeT commonsense embeddings per observation with GPT2, appending eighteen embeddings before Transformer layers.
  • Abductive Natural Language Generation: αNLG models generate hypothesis tokens conditioned on O1 and O2, optionally incorporating background knowledge K, and train by minimizing negative log-likelihood.

4 ART DATASET: ABDUCTIVE REASONING IN NARRATIVE TEXT

ART is a large-scale benchmark for abductive reasoning in narrative texts, pairing observations with explanatory hypotheses. It combines ROCStories observations with crowdsourced plausible and minimally edited implausible hypotheses, using adversarial filtering to reduce annotation artifacts.

  • ART contains ∼20K narrative contexts and over 200K explanatory hypotheses for studying abductive reasoning in narrative texts.
  • The best BERT-based model failed to correctly predict the first two examples in the ART development split.
  • Collecting Observations: The observation pairs O1 and O2 are drawn from ROCStories, whose five-sentence stories provide clear beginnings and endings.
  • Collecting Hypotheses Options: Workers generated plausible hypotheses by filling in what happened between the two narrative observations.
  • Collecting Hypotheses Options: Workers created implausible hypotheses by making up to five-word edits to plausible hypotheses while retaining the same observations.
  • Adversarial filtering retained challenging plausible–implausible pairs to reduce annotation artifacts that could leak target-label information.

5 EXPERIMENTS AND RESULTS

Experiments evaluate abductive inference and generation on ART using baseline and fine-tuned language models. Models remain substantially below human performance, while adversarial filtering and dataset design affect difficulty.

  • Abductive Natural Language Inference: Simple-feature baselines perform near chance, indicating that ART lacks simple annotation artifacts.The baselines test whether noticeable artifacts make the dataset trivially solvable.
  • Abductive Natural Language Inference: 58.8% accuracy from ESIM+ELMo, despite 88.9% on entailment NLI, shows that ART requires more than linguistic entailment.The comparison targets the related but distinct entailment task.
  • Abductive Natural Language Inference: 68.9% accuracy is achieved by BERT-ft, compared with 63.1% for GPT and 91.4% human performance on αNLI.BERT-ft uses the fully connected model and is the best-performing system.
  • Abductive Natural Language Inference: ~10,000 instances mark a performance plateau, leaving an approximately 23% gap between the best model and humans.The learning curve suggests that simply scaling the ART dataset may not close the observed gap.
  • Abductive Natural Language Inference: Adversarial filtering reduces GPT performance below 53%, while fully connected BERT reaches 72% versus 65% for the linear-chain BERT model.The results indicate that filtering disproportionately affects the model used as adversary and keeps ART challenging for other models.
  • Abductive Natural Language Generation: 45% is achieved by the best αNLG generators, compared with 96% for human-written hypotheses, even when models receive background commonsense knowledge.GPT2 is conditioned on both observations, with COMeT information integrated as textual phrases or embeddings.

6 ANALYSIS

The analysis examines where abductive models struggle across commonsense categories, implausible narrative transitions, and natural-language generation. Errors reflect both weak commonsense reasoning and difficulty producing plausible explanations.

  • Commonsense reasoning categories: 56.8% Numerical and 65.4% Spatial accuracy are BERT-ft’s weakest category results, while Emotional accuracy reaches 72.6%.The Emotional category contains strong textual cues about emotions and sentiments.
  • Implausible transitions: The negative hypothesis breaks at three possible narrative transitions: after O1, before O2, or through a less-plausible complete chain.These correspond to O1̸→h−, h−̸→O2, and a coherent but less plausible alternative.
  • Implausible transitions: Almost half of ART’s implausible transitions arise because h− is unlikely to precede O2, while all three transition categories are substantially represented.The proportions come from a subset of 1,000 test instances.
  • Abductive natural language generation: Generated-hypothesis examples include one instance only humans answer correctly and another where COMeT-Emb+GPT2 generates the correct explanation.Figure 6 compares trained-model generations with human-written hypotheses for two ART instances.

7 TRANSFER LEARNING FROM ART

ART is used not only as a benchmark but also for transfer learning to other commonsense datasets. Its benefits are strongest when target datasets contain relatively few training examples and disappear with abundant data.

  • Transfer learning from ART: Sequentially training on ART and then on WinoGrande, WSC, DPR, or HellaSwag improves performance over training only on the target dataset.The improvement is consistent on related datasets with relatively few training examples.
  • Transfer learning from ART: Pre-training on ART does not provide a significant improvement for target datasets with large amounts of training data.The transfer benefit therefore depends on the amount of target-dataset supervision.

8 RELATED WORK

Related work places ART between narrative event modeling, entailment datasets, commonsense inference, and generative language modeling. The paper distinguishes abductive reasoning from these settings by emphasizing plausible causal chains and non-monotonic commonsense inference.

  • Cloze-Style Task vs. Abductive Reasoning: Unlike script learning and narrative cloze tasks, this work reasons about the most plausible events conditioned on observations rather than learning prototypical narrative chains.The comparison frames abduction as explanation selection within narrative contexts.
  • Entailment vs. Abductive Reasoning: Abduction differs from entailment because it requires commonsense implications beyond linguistic entailment and non-monotonic reasoning.The paper identifies these as two critical distinctions making abductive reasoning uniquely challenging.
  • Generative Language Modeling: Generative language models produce reasonably coherent text but remain limited in non-monotonic generation and adherence to commonsense knowledge.These limitations motivate evaluating generation on abductive narrative explanations.
  • Related Datasets: ART complements existing resources that largely study deductive textual entailment or plausible-event inference in separate dataset settings.The related-dataset discussion positions ART as a resource for abductive reasoning in narrative text.

9 CONCLUSION

The paper introduces language-based abductive reasoning through two tasks and the ART challenge dataset. Results show substantial gaps between models and humans, especially for generating explanations.

  • Conclusion: ART contains 20,000 commonsense narratives and over 200,000 explanatory hypotheses supporting αNLI and αNLG.αNLI selects the more plausible hypothesis, while αNLG generates plausible hypotheses for observations.
  • Conclusion: 68.9% αNLI accuracy trails human performance of 91.4%, while the best αNLG generators reach 45% versus 96% for humans.The paper presents these results as evidence that abductive reasoning remains difficult for current language models.
  • Conclusion: The analysis identifies reasoning types that deep pre-trained language models fail to perform despite strong performance on closely related entailment NLI.These findings motivate further research on language-based abductive reasoning.

A.1 DATA COLLECTION DETAILS

The study collects plausible and implausible middle-sentence hypotheses for three-part stories, evaluates abductive hypothesis selection with human judgments and baselines, and uses adversarial filtering to construct difficult evaluation pairs.

  • Task 1 - Plausible Hypothesis Options: Task 1 asks participants to write a short, probable middle sentence explaining why observation O2 follows observation O1.The sentence must contain fewer than 10 words and narrate the likely causal connection between the observations.
  • Task 2 - Implausible Hypothesis Options: Task 2 minimally rewrites each plausible hypothesis into an unlikely, implausible, or inconsistent alternative.Participants may add or remove at most four words from the plausible hypothesis.
  • Task 3 - αNLI Human Performance: Human performance is measured by majority votes from seven participants choosing the more plausible hypothesis or “None of the above.”The evaluation presents an observation pair and two hypotheses for each question.
  • Dataset Construction: The ART train set contains all crowdsourced plausible and implausible hypotheses, whereas development and test sets use hypotheses selected by adversarial filtering.The filtering procedure selects one plausible and one implausible hypothesis that are hard to distinguish.
  • Baselines: 50.8% accuracy is achieved by the InferSent baseline, while SVM and bag-of-words models perform close to chance.The SVM uses simple lexical and sentiment features, and the bag-of-words model uses averaged GloVe embeddings.
Loading 1908.05739v2…