PaperlayerView exact revision on arXiv

Source-linked AI summary

"John is 50 years old, can his son be 65?" Evaluating NLP Models' Understanding of Feasibility

Himanshu Gupta, Neeraj Varshney, Swaroop Mishra, Kuntal Kumar Pal, Saurabh Arjun Sawant, Kevin Scaria, Siddharth Goyal, Chitta Baral

arXiv:2210.07471v2cs.CL

TL;DR

Feasibility reasoning—judging whether an action or its effect is possible—is a commonsense ability not covered by enough examples in existing reasoning datasets. The paper introduces FeasibilityQA, a dataset of binary and multi-correct questions, and evaluates language models on it. The tested models struggle, and providing relevant knowledge brings only a modest gain while performance remains low.

  • Problem

    Existing reasoning datasets do not provide ample examples testing whether actions or their effects are feasible, an ability relevant to commonsense reasoning about real-world situations.

    Source: §1 · p. 1

  • Method

    The authors introduce FeasibilityQA, about 5K feasibility questions in binary classification and multi-choice multi-correct formats, and evaluate models with and without relevant knowledge.

    Source: §1 · p. 2

  • Results

    GPT-3 achieves 19% MCQ and 62% BCQ accuracy zero-shot, and 25% MCQ and 64% BCQ accuracy few-shot; relevant knowledge improves performance by ∼7%, but overall performance remains low.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The results indicate that the evaluated models struggle to answer feasibility questions, including cases where GPT-3 generates a correct reasoning chain but gives the wrong answer.

    Source: §1 · p. 2

    The five selected categories are not an exhaustive coverage of feasibility.

    Source: §2.1.1 · p. 3, Limitations · p. 5

Abstract

from arXiv · show

In current NLP research, large-scale language models and their abilities are widely being discussed. Some recent works have also found notable failures of these models. Often these failure examples involve complex reasoning abilities. This work focuses on a simple commonsense ability, reasoning about when an action (or its effect) is feasible. To this end, we introduce FeasibilityQA, a question-answering dataset involving binary classification (BCQ) and multi-choice multi-correct questions (MCQ) that test understanding of feasibility. We show that even state-of-the-art models such as GPT-3, GPT-2, and T5 struggle to answer the feasibility questions correctly. Specifically, on MCQ and BCQ questions, GPT-3 achieves an accuracy of just (19%, 62%) and (25%, 64%) in zero-shot and few-shot settings, respectively. We also evaluate models by providing relevant knowledge statements required to answer the question. We find that the additional knowledge leads to a 7% gain in performance, but the overall performance still remains low. These results make one wonder how much commonsense knowledge about action feasibility is encoded in state-of-the-art models and how well they can reason about it.

1 Introduction

FeasibilityQA targets the underrepresented ability to judge whether actions or their effects are feasible, a commonsense reasoning challenge for language models. The authors introduce a dataset and find that tested models struggle, with only modest gains from added knowledge.

1 Introduction

  • Existing reasoning datasets cover many skills but lack ample examples testing feasibility understanding.The paper argues that evaluating this ability matters for developing NLP systems that can reliably reason about real-world situations.Source: §1 · pp. 1–2
  • FeasibilityQA contains about 5K instances in binary classification (BCQ) and multi-choice multi-correct (MCQ) formats, testing whether actions or their effects are feasible.BCQ asks whether a question is feasible given context; MCQ asks respondents to select all feasible answers.Source: §1 · pp. 1–2
  • GPT-3 achieves only 19% MCQ and 62% BCQ accuracy zero-shot, and 25% MCQ and 64% BCQ accuracy few-shot; added knowledge yields ∼7% improvement, but performance remains low.The authors also report that GPT-3 can generate a correct reasoning chain yet still fail to output the correct answer.Source: §1 · p. 2

2 FeasibilityQA

FeasibilityQA is built from real-life contexts and paired feasibility questions, with question creation and verification involving multiple student annotators. Its examples span five categories, while the dataset includes 1,536 contexts and 4,608 question pairs.

2.1 Dataset Creation

  • Seven computer science graduate students created 700 samples each over three months, followed by peer verification and cross-verification that rejected low-agreement samples.The paper notes that all dataset creators were also authors.Source: §2.1 · p. 2, §2.1.1 · p. 2

2.1.1 Context Creation

  • Contexts cover five categories: attribute comparison, implicit numerical reasoning, change with action, change with time, and non-numerical feasibility.The categories span topics including elementary physics, profit-loss scenarios, temporal comparisons, and quantity comparisons.Source: §2.1.1 · pp. 2–3
  • The authors describe the five selected feasibility categories as preliminary because the dataset cannot exhaustively cover feasibility.They identify expanding the category space as future work.Source: §2.1.1 · p. 3

2.1.2 Question Creation

  • Each real-life context is paired with two binary questions and one multi-correct multiple-choice question, with a None option available when no answer is feasible.The two BCQ questions use contrasting hypotheses, and answering both correctly for a context offers an additional consistency measure.Source: §2.1 · p. 2, §2.1.2 · p. 3

2.2 Dataset Statistics

  • The dataset contains 1,536 contexts and 4,608 context-question pairs: 3,072 BCQ and 1,536 MCQ, with BCQ labels balanced between True and False.MCQ questions vary in their number of correct options.Source: §2.2 · p. 3

3 Experiments

The experiments evaluate feasibility with exact-match, recall, and pairwise metrics, and find that models struggle across tasks and prompting settings. GPT-3 improves with added knowledge, but remains limited; category results and a case study further probe its errors.

3.1 Performance Evaluation & Metrics

  • For BCQ, pairwise accuracy requires both True and False questions for a context to be correct; MCQ exact match requires every correct option, while recall counts correctly predicted options.BCQ exact match is measured against each question’s True or False gold label.Source: §3.1 · p. 4

3.2 Results

  • GPT-3 reaches only 62.9% BCQ exact-match accuracy, while GPT-2 is at the 50% majority baseline and all models score very low on MCQ.The authors describe MCQ as more difficult than BCQ.Source: §3.2 · p. 4
  • In one-shot prompting, predictions are heavily influenced by the example’s label; chain-of-thought examples yield only marginal improvement.The one-shot pattern also appears in chain-of-thought experiments.Source: §3.2 · pp. 4–5
  • Knowledge improves GPT-3’s performance by ∼7% across settings, yet its accuracy remains 68.5% on BCQ and 37.2% on MCQ.The increase is ∼12% on few-shot MCQ.Source: §3.2 · p. 4
  • GPT-3’s MCQ recall reaches up to 70%, indicating it predicts some correct options but often misses other correct options.MCQ exact match requires predicting all correct options.Source: §3.2 · p. 4, §3.1 · p. 4
  • BCQ pairwise accuracy reaches at most ∼43%, versus ∼63% exact match; knowledge raises pairwise accuracy by ∼13%.The authors also report that knowledge narrows the gap between settings.Source: §3.2 · p. 4
  • In one-shot category results, the Non Numerical category performs best, by 15% over other categories, and knowledge improves categories by around 5% to 10%.The authors report similar category trends for MCQ.Source: §3.2 · p. 5, Table 7 · p. 5

A Case Study on Prompting GPT-3 to Provide Explanation:

  • In the explanation case study, GPT-3 did not understand the numerical value of Abraham’s age; even after receiving additional knowledge, it still could not understand the feasible age.Source: A Case Study on Prompting GPT-3 to… · p. 5

4 Conclusion

The paper concludes that FeasibilityQA evaluates understanding of feasibility and that tested models struggle even with additional relevant knowledge. It releases the dataset and calls for further research on feasibility reasoning as an underexplored aspect of commonsense reasoning.

  • FeasibilityQA is a question-answering dataset for evaluating understanding of feasibility.Source: §4 · p. 5
  • Across zero-, one-, and few-shot settings, tested models struggle to answer feasibility questions, and relevant additional knowledge yields only a small gain while overall performance remains low.Source: §4 · p. 5
  • The authors release the dataset and hope it encourages further research in feasibility reasoning, described as an important yet underexplored aspect of commonsense reasoning.Source: §4 · p. 5

Limitations

FeasibilityQA covers diverse real-world situations, but its category list is not exhaustive. The authors identify human evaluation and broader dataset coverage as future work.

Limitations

  • The dataset’s feasibility-situation list is explicitly non-exhaustive.The authors propose expanding categories using existing numerical datasets or by creating new category spaces.Source: Limitations · p. 5
  • Human evaluation is left for future work because it can be expensive.The authors also call for evaluators selected across educational backgrounds and ages, with a completely random selection for a comprehensive study.Source: Limitations · pp. 5–6
  • The dataset contains questions in English only.Source: Limitations · p. 6

Ethical Considerations

The dataset uses common English names and contexts designed to describe realistic situations. No personal information from data creators was collected.

Ethical Considerations

A Dataset characteristics

FeasibilityQA spans numerical and non-numerical reasoning categories, with examples illustrating how feasibility depends on relationships, changes, and contextual knowledge. Its topics range across several disciplines and 422 subcategories.

Dataset characteristics

Attribute Comparison

  • Attribute Comparison tests whether models infer comparative properties, such as a smaller bottle’s volume being below a larger bottle’s.Source: A Dataset characteristics · p. 8

Change with Time

  • Change with Time tests deductions about time-based changes, such as whether an age claim is possible given a person’s current age.Source: A Dataset characteristics · p. 8

Change with Action

Implicit Numerical Knowledge

Non Numerical

Dataset characteristics

B.1 Performance Metrics

The section describes true-positive, false-positive, and false-negative rates for MCQ evaluation. It also identifies tables reporting MCQ error rates and category-wise accuracy.

B.1 Performance Metrics

  • Table 9 reports GPT-3 false-positive rates and Table 10 reports its false-negative rates on MCQ.Source: Table 9 · p. 9, Table 10 · p. 9
  • Table 11 reports GPT-3 category-wise accuracy on BCQ and MCQ in the zero-shot setting.Source: Table 11 · p. 9
  • For MCQ, the section defines false negatives as incorrect predictions given as correct; its example counts an omitted option.Source: §B.1 · p. 9
  • For MCQ, the section defines false positives as correct predictions given as incorrect; its example identifies an extra predicted option.Source: §B.1 · p. 9

B.2 Results

Feasibility performance varies by task, category, and prompting setting, and knowledge affects error rates differently. GPT-2 reaches 50% accuracy because it predicts “False” in every case on the label-balanced dataset.

B.2 Results

  • False positive rates decreased with knowledge, while false negative rates decreased from zero-shot to few-shot but increased with knowledge in almost all cases.Source: §B.2 · p. 9
  • For zero-shot BCQ, Attribute Comparison had the lowest accuracy and Non-Numerical the highest; knowledge significantly improved Non-Numerical performance.Source: §B.2 · p. 9
  • Above 90% accuracy was reached for the first time in any category by few-shot BCQ in the knowledge setting for Non-Numerical questions.Attribute comparison remained the lowest-accuracy category, while Non-Numerical was the highest.Source: Table 12 · p. 9, §B.2 · p. 9
  • Few-shot MCQ with knowledge significantly improved in Non-Numerical, Change with Action, and Change with Time categories.Source: Table 12 · p. 9, §B.2 · p. 9
  • In zero-shot MCQ, Non-Numerical remained the best-performing category, with Change with Action producing similar results and no significant knowledge improvement in either category.Source: §B.2 · p. 9
  • 50% accuracy resulted from GPT-2 predicting “False” in all cases, incorrectly classifying every True hypothesis question in the label-balanced dataset.Source: §B.2 · pp. 9–10

C Case study: Chain of Thought Reasoning Approach

The chain-of-thought examples show that GPT-3 can sometimes reason to a correct feasibility answer, but other examples expose difficulty applying or associating logic with numerical quantities. Supplying knowledge did not improve those cases much.

C Case study: Chain of Thought Reasoning Approach

  • In the evaporation example, GPT-3 reasoned that water level would decrease and identified quantities below 63, including 59 and 61, as correct answers.The example was used to demonstrate reasoning in a one-shot setting without providing knowledge.Source: C Case study: Chain of Thought… · p. 10, Table 13 · p. 11
  • GPT-3 used the correct logic in one example but failed to calculate that 1600 was double 758, and another example used incorrect logic for adding a negative number.Source: C Case study: Chain of Thought… · p. 10
  • With knowledge supplied, the situation did not improve much: GPT-3 interpreted the logic correctly but could not associate it with numerical quantities.The examples are presented as a one-shot chain-of-thought setting with knowledge.Source: C Case study: Chain of Thought… · p. 10, Table 14 · p. 11

Related papers

Loading 2210.07471v2…