PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Fine-grained Causal Reasoning and QA

Linyi Yang, Zhen Wang, Yuxiang Wu, Jie Yang, Yue Zhang

arXiv:2204.07408v1cs.CLcs.AIcs.LO

TL;DR

Fine-grained causal relations such as enable and prevent are important in NLP but have been relatively overlooked. The paper constructs FCR, a human-labeled dataset for causal detection, extraction, and QA; experiments show that current models still have substantial room to improve, including a gap to human performance on fine-grained reasoning.

  • Problem

    Fine-grained causal perspectives such as enable and prevent have received limited attention despite their relevance to language understanding and NLP applications.

    Source: §1 · p. 1

  • Method

    The paper constructs FCR from human-labeled fine-grained event-causality pairs and defines causality detection, event-causality extraction, and Causal QA tasks.

    Source: §1 · p. 2, §4 · p. 4, §4.2 · p. 5

  • Results

    Experiments show room for improvement across causal-reasoning tasks; fine-grained task accuracy is 74.1% for models versus a 90.53% human ceiling.

    Source: §1 · p. 2, §6 · p. 8

  • Takeaways & Limitations

    The findings point to a need for better methods for event-causality correlation discovery and why/what-if QA.

    Source: §6 · p. 8

Abstract

from arXiv · show

Understanding causality is key to the success of NLP applications, especially in high-stakes domains. Causality comes in various perspectives such as enable and prevent that, despite their importance, have been largely ignored in the literature. This paper introduces a novel fine-grained causal reasoning dataset and presents a series of novel predictive tasks in NLP, such as causality detection, event causality extraction, and Causal QA. Our dataset contains human annotations of 25K cause-effect event pairs and 24K question-answering pairs within multi-sentence samples, where each can have multiple causal relationships. Through extensive experiments and analysis, we show that the complex relations in our dataset bring unique challenges to state-of-the-art methods across all three tasks and highlight potential research opportunities, especially in developing "causal-thinking" methods.

1 Introduction

Fine-grained causal relations such as enable and prevent matter for NLP applications, but have received limited attention. The paper introduces FCR, a human-labeled dataset and task suite, and reports a substantial gap between model and human performance on fine-grained reasoning.

1 Introduction

  • Fine-grained event relations matter for language understanding and applications including information extraction, question answering, and machine reading in high-stakes domains.Source: §1 · p. 1
  • The dataset distinguishes cause, enable, and prevent; enable is sufficient but not necessary, whereas cause is typically necessary and sufficient.Source: §1 · pp. 1–2
  • FCR contains 25,193 cause-effect pairs and 24,486 QA pairs concerning fine-grained causal relations.Almost all questions are why or what-if questions.Source: §1 · p. 2
  • FCR is presented as the first human-labeled fine-grained event-causality dataset, supporting causality detection, fine-grained extraction, and Causal QA.Source: §1 · p. 2
  • 74.1% model accuracy versus 90.53% human ceiling accuracy shows a substantial gap on the fine-grained task.The introduction contrasts this result with 94% F1 on traditional cause-effect detection.Source: §1 · p. 2

2 Related Work

Prior work addresses textual event-causality detection and causal question answering, but fine-grained relations and their combination with span-based QA remain limited. FCR joins these elements in a dataset built from human-labeled causal relations.

Fine-grained Event Causality

  • FinCausal [Mariko et al., 2020a] is described as a relatively small Edgar-derived dataset focused on cause relations and without QA tasks.Table 1 compares event-causality and QA datasets.Source: §2 · p. 2
  • Textual event-causality work has focused on graph-based and event-level detection, while fine-grained reasoning between events has received relatively little attention.Prior systems can extract a COVID-19 cause relation but miss further effects involving Amazon pricing and offline-business recruitment.Source: §2 · p. 2

Causal QA

  • Existing QA datasets generally emphasize factual questions, while FCR combines fine-grained human-labeled causality with span-based why and what-if questions.The related-work discussion identifies no compared dataset as combining causal reasoning and span-based QA.Source: §2 · pp. 2–3

3 Dataset

The FCR dataset is built from financial reports with human labels for causal relations and QA, and its multi-sentence, multi-relation samples pose challenges beyond simple causal detection.

3 Dataset

  • FCR uses 6,786 Yahoo Finance analyst reports published between December 2020 and July 2021, comprising 54,289 sentences.Each report concerns a U.S.-listed company.Source: §3 · p. 3, §3.1 · p. 3

3.1 Crowdsourcing

  • Annotators labeled causal presence, cause-effect spans and relation types, then generated QA pairs from the labeled event triples.Relation labels include cause, enable, prevent, their inverse forms, and irrelevant; active relations typically yield why-questions and passive relations what-if questions.Source: §3.1 · p. 3
  • Quality checks revised roughly 3% of instances, and inter-annotator agreement reached 91% for fine-grained labels and 94% F1 for causal QA pairs.Labelers with error rates above 10% were excluded from the second annotation round.Source: §3.1 · pp. 3–4
  • The dataset has 51,025 instances, including 21,046 with at least one causal relation, and is chronologically sorted to keep future data out of prediction training.Chronological ordering is intended to make future test data less likely to overlap with training data.Source: §3.1 · p. 4

3.2 Discussion

  • It includes 846 multi-sentence samples and 3,017 text chunks with multiple causal relations, while positive and negative examples have similar average token counts in Tasks 1 and 2.The authors note that multi-relation samples require complex reasoning and that similar token counts limit shortcut cues from instance length.Source: §3.2 · p. 4

3.3 Meta-information

  • Company names and publication dates provide metadata; Consumer Cyclical, Industrial, and Technology are the largest sectors, while Utilities is the smallest.The authors use the three largest sectors for out-of-domain evaluation and company names in question templates.Source: §3.3 · p. 4

4 Tasks and Methods

The experiments benchmark three FCR tasks, progressing from detecting causal text to extracting fine-grained event relations and answering why and what-if questions. The methods use neural and Transformer-based models for prediction and generative QA.

4 Tasks and Methods

  • The experiment pipeline benchmarks causal detection, fine-grained event causality extraction, and causal question answering.The paper presents these as three tasks on the FCR dataset.Source: §4 · p. 4, §4.1 · p. 5, Figure 4 · p. 5

4.1 Causal Detection

  • Task 1 is binary classification of whether a text sequence contains a causal relation.Multi-sentence samples are included because causality can span them.Source: §4.1 · p. 5

4.2 Fine-grained Event Causality Extraction

  • Task 2 extracts cause and effect spans as a multi-span event task, assigning each event pair one of seven causal labels.The reported experiment uses 33,634 event triples; detected spans are reserved for generating causal questions.Source: §4.2 · p. 5

4.3 Causal QA

  • For extractive QA, the context and question are concatenated with special tokens, and a pretrained Transformer predicts the answer span boundaries.The input sequence uses [CLS] and [SEP] markers.Source: §4.3 · p. 5

4.4 Models

  • Predictive baselines include CNN-Test, HAN, BERT, RoBERTa, and SpanBERT; generative QA benchmarks use T5-small/base/large and BART-base/large.The methods also include BERT and RoBERTa variants for causal reasoning QA.Source: §4.4 · pp. 5–6

5 Experiments

Experiments evaluate causality detection, fine-grained event extraction, and Causal QA with held-out splits and pretrained models. Results show gaps from human performance, difficulty on joint and complex causal reasoning, and mixed effects from additional QA training data.

5 Experiments

  • The experiments report results for three tasks based on the FCR dataset.The section describes quantitative and qualitative analyses for Causal QA.Source: §5 · p. 6, §5.4 · p. 6

5.1 Settings

  • The benchmark uses mutually exclusive 8:1:1 train, validation, and test splits across tasks, with validation-set hyperparameter tuning.It reports Macro F1 and accuracy for event causality, and exact match and F1 for Causal QA.Source: §5.1 · p. 6

5.2 Causality Detection

  • RoBERTa-Large’s 84.64 F1 for causality detection remains below human performance of 94.32.The paper contrasts this fine-grained task with FinCausal, where the best reported result reaches human-level F1 of 97.75.Source: §5.2 · p. 6

5.3 Fine-grained Event Causality Extraction

  • SpanBERT and RoBERTa achieve the best reported extraction and fine-grained classification scores, but the joint task’s highest test EM is only 21.78.The joint task requires event extraction and classification to match the ground truth exactly; RoBERTa-Large also performs better on simple relations than on Enable and Enable_by, while Prevent_by is a low-example category.Source: §5.3 · p. 6
  • BERT-large does not improve fine-grained classification over BERT-base: their test results are 60.24 versus 63.72 F1 and 69.85 versus 71.72 accuracy.The comparison uses BERT-base and BERT-large on fine-grained classification.Source: §5.3 · p. 6

5.4.1 Quantitative Analysis

  • RoBERTa-large reaches 84.35 F1 on Causal QA; SQuAD2.0-only RoBERTa-large scores 63.82 versus 84.35 on FCR, while additional SQuAD2.0 training reaches 84.65.T5-Large’s F1 of 81.77 is comparable to RoBERTa-large’s 84.35, while generative methods’ average EM is below that of extractive methods using the same training data.Source: §5.4.1 · p. 7

5.4.2 Challenges by Causal QA

  • The paper compares best-performing models across QA datasets, reporting 61.6 EM for Causal QA versus 90.9 on SQuAD2.0 and 39.3 versus 85.6 accuracy for LogiQA and FCR.The authors say LogiQA is more challenging because it requires heavy logical reasoning rather than identifying causal relations from text.Source: §5.4.2 · p. 7

5.4.3 Error Analysis

  • In two qualitative error examples, the model misses a gold answer by confusing repeated words or by choosing an answer near the question instead of using the whole sentence.The examples include overlooking distant reasons identified as “equity and credit markets” and “Brexit.”Source: §5.4.3 · p. 8

6 Conclusion

The conclusion reports that current state-of-the-art pretrained models leave room for improvement on causal reasoning tasks. It calls for better solutions to event-causality discovery and Why/What-if QA.

6 Conclusion

  • The paper’s evaluation uses a novel fine-grained reasoning setting with rich human labels.The conclusion presents this setting as the basis for exploring current methods’ efficacy.Source: §6 · p. 8
  • Experiments with state-of-the-art pretrained language models indicate substantial room for improvement on causal reasoning tasks.The paper frames its evaluation around a fine-grained reasoning setting and a dataset with rich human labels.Source: §6 · p. 8
  • The conclusion identifies event-causality discovery and Why/What-if QA as areas needing better solutions.These are the paper’s named causal reasoning task areas.Source: §6 · p. 8

7 Ethical Statement

The ethical statement describes the data sources, annotator compensation, and release license.

7 Ethical Statement

  • The study uses publicly available financial analysis reports and says it uses no private data or non-public information.The statement says the paper honors the ACL Code of Ethics.Source: §7 · p. 8
  • Annotators received labor fees corresponding to the amount of corpus they annotated.Source: §7 · p. 8
  • The paper’s code and data are released under the CC-BY-NC-SA license.Source: §7 · p. 9

A Appendix: Annotation Instructions

The appendix describes the annotation platform and instructions for labeling event causality. Annotators identify causal sentences, mark events, and assign Cause, Enable, or Prevent relations, with examples illustrating negative, positive, and multi-event cases.

A Appendix: Annotation Instructions

A.1 General Instruction

  • The general instruction asks annotators to find all cause-effect event pairs and their fine-grained relation types in passages.Source: §A.1 · p. 13

A.2 Steps

  • A sentence is positive when it contains at least two causally related events; otherwise, it is negative.For positive sentences, annotators identify events and pair those that form causal relations.Source: §A.2 · p. 13
  • The three relation labels are Cause, Enable, and Prevent, with Enable described as a possible but non-necessary way for an event to happen.The instruction defines Cause as B always happening if A happens, and Prevent as A and B being unable to happen at the same time.Source: §A.2 · p. 13
  • Annotators are told to mark every event-causality pair before moving to the next passage.Source: §A.2 · p. 13

A.3 Examples

  • The examples include a negative sentence with no causal relation and a positive sentence labeled Cause.The examples also show that a sentence with four events requires checking the possible event pairs.Source: §A.3 · p. 13

B Appendix: Out-of-domain Test

The appendix tests out-of-domain generalization using instances from three dataset sectors with the largest sample counts. Both tasks show cross-domain performance decay, and same-domain train/test pairs perform best.

B Appendix: Out-of-domain Test

Loading 2204.07408v1…