PaperlayerView exact revision on arXiv

Source-linked AI summary

CRASS: A Novel Data Set and Benchmark to Test Counterfactual Reasoning of Large Language Models

Jörg Frohberg, Frank Binder

arXiv:2112.11941v3cs.CL

TL;DR

Evaluating LLM progress on common-sense reasoning beyond quickly solved benchmarks remains an ongoing challenge. CRASS uses questionized counterfactual conditionals, builds a crowd-validated dataset, and benchmarks six models. The tested models leave a gap of over 25% compared with the human baseline, indicating room for improvement.

  • Problem

    Evaluating LLM progress on common-sense reasoning beyond established benchmarks that are being solved quickly remains an ongoing research challenge.

    Source: §1 · p. 1

  • Method

    CRASS builds a dataset of questionized counterfactual conditionals, validates items with crowd workers, establishes a human baseline, and benchmarks six state-of-the-art models.

    Source: §1 · p. 1, §2.4 · p. 2, §3 · p. 2

  • Results

    The tested LLMs leave a gap of over 25% compared with the human baseline.

    Source: §1 · p. 1

  • Takeaways & Limitations

    The authors conclude that CRASS poses a valid challenge for tested models and that their results leave considerable room for improvement.

    Source: §4.2 · p. 4, §7 · p. 6

    CRASS tests a specific reasoning ability framed as a counterfactual question, which may introduce encoding artifacts or overfitting if models are pretrained on many QCCs.

    Source: §5.3 · p. 5

Abstract

from arXiv · show

We introduce the CRASS (counterfactual reasoning assessment) data set and benchmark utilizing questionized counterfactual conditionals as a novel and powerful tool to evaluate large language models. We present the data set design and benchmark that supports scoring against a crowd-validated human baseline. We test six state-of-the-art models against our benchmark. Our results show that it poses a valid challenge for these models and opens up considerable room for their improvement.

1. Introduction

CRASS addresses the ongoing need to evaluate progress on common-sense reasoning beyond benchmarks that are being solved quickly. It introduces questionized counterfactual conditionals as a benchmark and reports a gap of over 25% between tested LLMs and the human baseline.

1. Introduction

  • CRASS introduces questionized counterfactual conditionals as a new scheme for evaluating LLM reasoning and natural-language understanding.The benchmark was accepted as part of the BIG-bench suite.Source: §1 · p. 1
  • The benchmark addresses the ongoing need to evaluate progress beyond established benchmarks such as SuperGLUE, which are being solved increasingly quickly.Source: §1 · p. 1
  • The tested LLMs leave a gap of over 25% compared with the human baseline.Source: §1 · p. 1

2. Basic Concepts and Theoretical Background

The paper adopts narrow working meanings for conditionals, counterfactuals, causes, effects, and events, then uses those assumptions to formulate questionized counterfactual tests. A QCC asks what would have happened under an altered antecedent and scores candidate consequences.

2. Basic Concepts and Theoretical Background

  • For this paper, a conditional is an if-then clause relating an antecedent and consequence, causality is treated agnostically, and an event is a singular dynamic event token.Source: §2 · p. 1

2.1. Counterfactual Conditionals

  • A counterfactual conditional links a consequence to an antecedent known to be false, representing a causing event that did not happen but could have.Using past perfect tense lets the authors ask counterfactual questions about a terminated event without violating the chain of events.Source: §2.1 · pp. 1–2

2.2. Causal Model

  • The causal model assumes a causal structure relating events and a mechanism supporting counterfactual reasoning; selecting the correct consequence passes the benchmark test.The paper assumes a cause-effect asymmetry, with p(E|C) consistent with the world’s causal model but p(C|E) not.Source: §2.2 · p. 2

2.3. Premise Explication

  • Counterfactual conditionals carry an implicit base premise, and their linguistic antecedents and consequences should not be equated directly with causes and effects.The paper illustrates a basic premise with a noun phrase, verb, and preposition linking two nouns.Source: §2.3 · p. 2

2.4. Questionized Counterfactual Conditionals (QCC)

  • A QCC asks what would have happened if an altered, counterfactual antecedent had held, then selects the most probable suitable consequence.The process starts from a randomly selected causally relevant base premise and forms a competing, obviating antecedent.Source: §2.4 · p. 2

3. Data set Design

CRASS items pair a base premise and questionized counterfactual with a correct consequence and distractors. The authors generated and crowd-validated items, retaining 274 high-quality premise-counterfactual tuples, including impossible scenarios in a small portion of the set.

3. Data set Design

  • The authors began with about 90 AMT gold-standard examples, expanded from ATOMIC’s xEffect category to 708 items, then used AMT ratings to filter the set.A premise-counterfactual tuple combines a base premise and QCC with a correct consequence and potential-effect distractors.Source: §3 · pp. 2–3
  • 274 high-quality premise-counterfactual tuples remained after items rated below 4 were excluded; 44 were contributed to BIG-bench.Each tuple offers three possible answers, including the correct consequence and distractors.Source: §3 · p. 3
  • 6% of examples pose impossible situations under currently observable physics and other laws of nature.The authors note that unrealistic scenarios challenge existing language models.Source: §3 · p. 3
  • Generating and validating high-quality language data is costly, constraining CRASS’s size and limiting it to English.The authors say translation and provision in other languages were too costly at the time.Source: §3 · p. 3

4. Benchmark

The benchmark evaluates zero-/few-shot and NLI-fine-tuned models on counterfactual questions, using top-1 accuracy and a crowd-validated human reference. Human annotators achieved 0.98 top-1 accuracy across the PCTs.

4.1. Setup and Model Selection

  • Four NLI-fine-tuned classifiers and the general-purpose models GPT-3 and T0pp were benchmarked; poor zero-shot output led to few-shot prompting for these multi-purpose models.The classifiers included models trained on MNLI, with DeBERTa also trained on SNLI and ANLI.Source: §4.1 · p. 3

4.2. Evaluation and Metrics

  • Top-1 accuracy counts the model’s single highest-probability answer as correct or incorrect against the total answers.For multi-purpose models, answers were repeated 10 times to derive prediction scores; the study also reports average confidence and its range.Source: §4.2 · pp. 3–4
  • T0pp picked up the required task in zero- or one-shot settings more often than GPT-3.Source: §4.2 · p. 3
  • 0.98 top-1 accuracy was the human baseline across all PCTs, providing a crowd-validated reference for model performance.The final study used 13 selected annotators; each PCT was scored by at least seven annotators.Source: §4.2 · p. 4

5. Benchmark Results

The tested models scored below the human baseline on CRASS, with weaknesses on impossible situations and sensitivity to some answer-option changes. The paper also discusses possible sources of these difficulties and limits the benchmark to a specific counterfactual-question framing.

5.1. Results and Discussion

  • T0pp reached the best T1 at 72.63%, while the human baseline outperformed every model by at least 25 percentage points in the best case and 63.15 in the worst.The evaluated classifiers included BART, RoBERTa, MPNet, and DeBERTa; GPT-3 and T0pp were also tested.Source: §5.1 · p. 4
  • All models struggled with impossible-situation QCCs, where T0pp and GPT-3 each reached 31.25% T1, while RoBERTa reached 56.25%.The paper suggests NLI training, unrealistic scenarios, and the transformer architecture’s word-prediction focus as possible reasons for poor performance.Source: §5.1 · pp. 4–5
  • Attribution for BART on PCT #56 suggests focus on the bridge while neglecting the phrase signaling the hypothetical event.Source: §5.1 · p. 5, Figure 2 · p. 5

5.2. Robustness

  • Only about 14% of predictions changed after answer labels were altered; BART retained 91.3% T1, versus 78.3% for DeBERTa and 82.6% for MPNet.The robustness check changed the form or syntax of an agreed-upon answer while keeping its meaning unchanged; all six models agreed on the correct answer for 23 items.Source: §5.2 · p. 5, Table 2 · p. 5

5.3. Limitations

  • CRASS tests a specific reasoning ability through counterfactual questions, a framing that may introduce encoding artifacts or overfitting if models are pretrained on many QCCs.Source: §5.3 · p. 5

6. Related Works

The related-work discussion situates CRASS among commonsense benchmarks and several lines of counterfactual research. It distinguishes dynamic reasoning about consequences from static world-knowledge tests and notes connections to causal and explanatory approaches.

6.1. Benchmarks

  • CRASS is positioned beyond static commonsense tests, such as asking whether a tuba fits in a backpack, because it targets reasoning and inference in commonsense statements.The paper also reviews newer benchmarks including WinoWhy, COM2SENSE, UNICORN, and CommonSenseQA 2.0.Source: §6.1 · pp. 5–6

6.2. Counterfactuals

  • Related counterfactual work includes image-based alternatives, structural causal models, and applications in network and graph models.The paper notes that merely removing an object from an image is not a counterfactual; an alternative object is needed, and SCMs may provide causal nets relevant to QCCs.Source: §6.2 · p. 6
  • Counterfactual explanations seek data-point changes that alter predictions, while causal-learning work is discussed as a possible source of structured causal information for LLMs.The paper proposes potential value in combining causal-learning research with methods for extracting structured causal data from language corpora.Source: §6.2 · p. 6

7. Conclusion

The paper presents CRASS as a reasoning and language-understanding dataset, benchmarks state-of-the-art models, and establishes a human baseline. It reports large margins for future LLM improvement.

7. Conclusion

  • The paper’s contributions are the CRASS dataset, analysis of state-of-the-art models’ challenges, and an extensive crowd-worker human baseline; results show large margins for future LLM improvement.Source: §7 · p. 6

10. Language Resource References

CRASS is referenced as a versioned dataset available through GitHub, with preservation measures and an API for accessing data and returning model answers.

10. Language Resource References

  • The CRASS dataset is listed as version 1.0 in its language-resource reference.Source: §10 · p. 10

A.1. Data Set Location and Availability

  • The dataset is available from a GitHub repository.Source: §A.1 · p. 11

A.3. Data Preservation and Maintenance

  • Long-term availability and data fetching are supported through GitHub practices and the GitHub Archive Program.Source: §A.3 · p. 11

A.4. Canary String

  • A canary string is included to discourage Big-bench models from training on the dataset.Source: §A.4 · p. 11

B. CRASS API

  • The CRASS API provides dataset retrieval by GET request and accepts model answers as JSON by POST request after model registration.Source: B. CRASS API · p. 11

C. Model Configuration and Computational Resources

This material documents model configuration, counterfactual-generation categories, a few-shot workaround for multi-purpose models, and the AMT study used to establish a human baseline.

C. Model Configuration and Computational Resources

  • Models were tested zero-shot without fine-tuning; fixed-seed hosted classifiers reproduced results, and inference took 544.08 seconds.Tests ran between October 10 and November 23, 2021.Source: C. Model Configuration and… · pp. 11–12

D. Classification of counterfactual generation

  • Seven generation methods include negation and single-component exchanges, while complex variants combine components or swap subject and object.Some preposition exchanges require prior knowledge of spatial relations and location to classify and answer the questions.Source: D. Classification of counterfactual… · p. 12

E. Multi-Purpose Model Few-Shot Prompt

  • For GPT-3 and T0pp, three examples cover standard, reversed-causality, and impossible cases; repeated calls estimated confidence because the classification endpoint lacked confidence scores.The prompt workaround followed GPT-3's requirement for examples and used at least ten repetitions.Source: E. Multi-Purpose Model Few-Shot Prompt · pp. 12–13

F. Crowd Worker Platform Statement

  • The AMT qualification survey drew 73 annotators, 21 were selected, and 11 participated in the final study; the main-study rate was $30.30 per hour.Including qualification, the reported average was $25.19 per hour.Source: F. Crowd Worker Platform Statement · pp. 13–14

G. Robustness of Models

The robustness analysis changed the wording of unanimously correct answers while preserving meaning; all six models had agreed correctly on about 8.4% of dataset items.

G. Robustness of Models

Related papers

Loading 2112.11941v3…