Source-linked AI summary
A Survey on Extraction of Causal Relations from Natural Language Text
Jie Yang, Soyeon Caren Han, Josiah Poon
TL;DR
Cause-effect relations are important across applications, yet surveys focused specifically on causality extraction are scarce. This paper reviews causal forms, datasets, metrics, and extraction approaches; its reviewed benchmark results show deep-learning methods often achieve state-of-the-art performance, with an exception on BioInfer.
Problem
Method
The paper surveys causal forms, six benchmark datasets, evaluation metrics, representative systems, and open challenges across traditional and modern extraction approaches.
Source: §8 · p. 22Results
Deep-learning methods achieve state-of-the-art results and substantial improvements on most benchmark datasets, except, which records 49.8% F-score on BioInfer.
Source: §6 · p. 18Takeaways & Limitations
The review concludes that domain-specific pre-trained models combined with graph models hold potential for causality extraction.
Source: §8 · p. 22TACRED has only 269 cause-of-death instances available for the causality-extraction task.
Source: §3 · p. 6
Abstract
from arXiv · showhide
As an essential component of human cognition, cause-effect relations appear frequently in text, and curating cause-effect relations from text helps in building causal networks for predictive tasks. Existing causality extraction techniques include knowledge-based, statistical machine learning(ML)-based, and deep learning-based approaches. Each method has its advantages and weaknesses. For example, knowledge-based methods are understandable but require extensive manual domain knowledge and have poor cross-domain applicability. Statistical machine learning methods are more automated because of natural language processing (NLP) toolkits. However, feature engineering is labor-intensive, and toolkits may lead to error propagation. In the past few years, deep learning techniques attract substantial attention from NLP researchers because of its' powerful representation learning ability and the rapid increase in computational resources. Their limitations include high computational costs and a lack of adequate annotated training data. In this paper, we conduct a comprehensive survey of causality extraction. We initially introduce primary forms existing in the causality extraction: explicit intra-sentential causality, implicit causality, and inter-sentential causality. Next, we list benchmark datasets and modeling assessment methods for causal relation extraction. Then, we present a structured overview of the three techniques with their representative systems. Lastly, we highlight existing open challenges with their potential directions.
1 Introduction
The introduction frames causality extraction as identifying cause-effect relations between entities, important in applications such as medicine. It distinguishes approaches by technique and causal form, noting that implicit and inter-sentential cases are more complex than basic forms.
1 Introduction
- A cause-effect relation links two entities when the occurrence of one results in the occurrence of the other, with medical uses including treatment decisions and knowledge-graph construction.The paper cites clinical decision-making and biomedical knowledge discovery as application areas.Source: §1 · p. 2
- Causality extraction systems use knowledge-based, statistical machine-learning, or deep-learning approaches, and address explicit or implicit relations within or across sentences.Explicit cues include causal connectives, causative verbs, resultative phrases, and conditional constructions; implicit cases may have ambiguous or no connectives.Source: §1 · p. 3
- Many existing approaches target explicit and/or intra-sentential causality, while implicit and/or inter-sentential cases are described as more complicated.Source: §1 · p. 3
2 Previous surveys
The paper identifies a scarcity of surveys focused specifically on causality extraction. It contrasts causality extraction with relation extraction and notes that earlier reviews did not cover recent deep-learning developments.
2 Previous surveys
- Surveys focused specifically on causality extraction are notably scarce, while existing reviews often cover relation extraction more broadly.Source: §2 · pp. 3–4
- Causality extraction differs from relation extraction in its binary rather than multiple-classification framing and in the linguistic patterns or features used.The passage contrasts causal relations' defined target with relation extraction's possible open-domain relation types.Source: §2 · p. 4
- Earlier surveys reviewed handcrafted, semi-automatic, and supervised approaches, but lacked coverage of recent developments, especially deep learning.Asghar compared non-statistical and statistical/ML techniques; Barik et al. grouped methods into four categories.Source: §2 · p. 4
3 Benchmark datasets
The paper surveys six causality datasets: four from general domains and two biomedical. It describes their sources, size, availability, and class balance, while noting limited examples in some benchmarks.
3 Benchmark datasets
- The survey covers six datasets—four general-domain and two biomedical—and compares their causality sizes, sources, availability, and class balance.Source: §3 · pp. 5–6, Table 3 · p. 6
- SemEval-2007 Task 4 includes 140 training and 80 test Cause-Effect examples, with positive proportions of 52.0% and 51.0%, respectively.The authors identify its small data amount as a limitation.Source: §3 · p. 5
- SemEval-2010 Task 8 is a nine-relation multi-classification dataset with 10,717 annotated examples, and its small, imbalanced sample is a major limitation.The passage reports 1,003 training examples with 13.0% positive data and 328 test examples with 12.0% positive data.Source: §3 · p. 5
- PDTB 2.0 contains 72,135 non-causal and 9,190 causal examples from 2,312 Wall Street Journal articles, including implicit AltLex relations.The authors note that its complex storage format requires conversion tools.Source: §3 · p. 5
- TACRED provides only 269 cause-of-death instances for causality extraction, which the paper identifies as its main limitation.Source: §3 · pp. 5–6
- The biomedical examples include BioInfer, with 1,100 sentences and 1,461 causal-effect relations, and ADE data drawn from 1,644 PubMed abstracts.The BioInfer passage reports 2,662 relations total; the ADE passage describes 6,821 sentences with at least one adverse drug effect relation.Source: §3 · p. 6
4 Evaluation metrics
The paper reviews commonly used classification metrics and other measures tailored to particular extraction tasks. It cautions that accuracy and F-score can mislead on imbalanced data and describes alternatives that account for both classes.
4 Evaluation metrics
- For causal-pair classification, true positives, false positives, true negatives, and false negatives count correctly or incorrectly identified causal and non-causal pairs.Accuracy and F-score are among the most commonly used classification metrics.Source: §4 · p. 7
- Accuracy and F-score may overstate performance on imbalanced datasets, whereas MCC treats both classes equally and G-mean reflects performance balance across classes.The paper notes that weak positive-class prediction lowers G-mean even when negative instances are correctly classified.Source: §4 · p. 7
- Entity-labeling evaluation can assess whether models identify cause and effect boundaries, including F-scores for cause, effect, connective, and none tags.The passage contrasts labeling evaluation with relation extraction because causes and effects may span multiple words without entity masks.Source: §4 · pp. 7–8
- Why-question answering studies use P@N and MAP, while causality-topic mining uses causality confidence and topic purity.The passage defines P@N through correct answers among top-N passages and MAP through the ranking quality of answer passages.Source: §4 · p. 8
5 Causal relation extraction methods
The survey classifies causality-extraction systems by technique and by whether causal relations are explicit or implicit and intra- or inter-sentential. It reviews representative knowledge-based, statistical machine-learning, and deep-learning methods across these forms.
5.1 Knowledge-based approaches
- Knowledge-based causal extraction uses predefined patterns or rules; structural analysis improves performance and is more likely to extract implicit causality, while COATIS reaches 85.2% precision on small French fragments.Pundit reports 70.4% precision on news titles spanning 150 years, with 10.0% recall.Source: §5.1 · pp. 8–9
- Ittoo and Bouma acquire three implicit-causality pattern types from Wikipedia, while Khoo et al. use verbal patterns to extract within- and between-sentence relations.Khoo et al. report 68.0% accuracy on 1,082 Wall Street Journal sentences; errors often reflect complex syntax and limited world-knowledge inference.Source: §5.1 · pp. 9–10
5.2 Statistical machine learning-based approaches
- For explicit intra-sentential patterns, Blanco et al. report 91.3% F-score on TREC, while three SemEval-2010 Task 8 models report F-scores of 85.8%, 73.9%, and 85.6%.The survey notes that errors often involve the relators “as” or “after”; Lin et al. report 51.0% F-score on PDTB 2.0, with high recall but very low precision for causality.Source: §5.2 · p. 11
- Hidey and McKeown use PDTB 2.0 AltLex connectives as seed data to find new alternative lexicalizations, reporting a 75.3% F-score and an 11.1% improvement over baseline.Other approaches use language- or domain-specific features, including Arabic linguistic features and verb-pair rules for Thai agriculture texts.Source: §5.2 · p. 12
- Inter-sentential systems use lexical pairs, cue phrases, or heuristic entity-pair constraints; Qian and Zhou apply maximum-entropy models to intra- and inter-sentential chemical-induced disease relations.Their inter-sentence instances exclude entity pairs already involved in intra-sentential instances, require sentence distance under three, and retain shortest-distance pairs when multiple entities occur.Source: §5.2 · p. 13
5.3 Deep learning-based approaches
- Deep models use architectures including LSTMs, attention-based CNNs, and dependency-tree GCNs; the survey reports attention gains for long-distance causality and cascading errors from dependency parsing.For SemEval-2010 Task 8, multi-head self-attention helps when causal distance exceeds 10; the GCN's F-score is 3.2% below the attention CNN's.Source: §5.3 · p. 14
- On implicit causality, the word-embedding-only LSTM reports 81.9% F-score on PDTB 2.0 AltLex, while ADE joint model CMAN reaches 81.1%, 1.0% above.The survey also describes reinforcement learning to relabel noisy instances and joint models that connect entity and relation representations.Source: §5.3 · pp. 15–16
- For intra- and inter-sentential event causality, a multi-column CNN combines candidate, context, and background-knowledge inputs; related background knowledge significantly improves performance.The columns have independent convolutional and pooling layers, and their outputs are concatenated for classification.Source: §5.3 · p. 16
6 Systems summary
Across 45 reviewed systems, the survey compares causality forms and technique families, finding that deep learning leads on most benchmark datasets while method choice depends on local needs and context.
Reviewed systems
- Of 45 reviewed systems, 50.0% target explicit intra-sentential causality, while 43.2% use deep learning.The other reviewed forms are implicit (31.8%) and inter-sentential (18.2%); techniques also include statistical ML (36.4%) and knowledge-based methods (20.4%).Source: §6 · p. 16, Figure 4 · p. 17
Technique families
- Knowledge-based approaches use predefined linguistic rules or patterns, making explicit causal connectives useful clues.They require more elaborate clues when connectives are absent, and preparing varied clues is time-consuming; the survey says they are unsuitable when dataset consistency is poor.Source: §6 · pp. 16–17
- Statistical ML systems use linguistic features or designed kernels, but well-prepared features or kernels may weaken portability downstream.For explicit intra-sentential causality, classifiers use manually or automatically collected features to remove non-causal instances; missing keywords can require more complex features.Source: §6 · p. 17
- Deep learning systems automatically infer higher-level information from input vectors, improving portability across applications.They require larger corpora and more computational resources than the other two technique families.Source: §6 · p. 17
Technique choice
- The survey recommends considering specific needs and contextual aspects when choosing a causality-extraction method.Figure 5 presents the process for choosing an appropriate technique.Source: §6 · p. 17, Figure 5 · p. 18
Evaluation datasets
- The remaining 17 approaches evaluate model performance using self-collected datasets or other publicly available datasets; these approaches are summarized in Table 6.Source: §6 · p. 18
Benchmark results
- On SemEval-2010 Task 8, Kyriakakis et al. surpass by 16.7% F-score; on ADE, exceeds by 25.8%.Table 5 covers approaches on benchmark datasets; a BioInfer model is an exception, with 49.8% F-score, 11.5% below, which the survey connects to noisy data.Source: §6 · p. 18, Table 5 · p. 19
7 Open problems and future directions
The survey identifies multiple causalities, data deficiency, and document-level extraction as open problems, and reviews approaches that address their distinct demands.
Multiple Causalities
- Most prior systems extract one causal pair per instance, although real-world text includes domino, relational, and mutual causalities.Splitting sentences into sub-sentences to extract pairs separately is computationally expensive and misses dependencies; Tag2Triplet generates multiple triplets, while deep models use relational reasoning to identify multiple relations simultaneously.Source: §7 · pp. 19, 21
Data Deficiency
- Annotated real-world data is specific and small relative to deep-learning needs, and distant supervision can create noisy labels.The survey describes approaches for incomplete or incorrect labels and a hybrid approach that encodes sentences with an attention-based BiLSTM, then learns relation types with a random forest.Source: §7 · p. 21
Document-level Causality
- Document-level causality can span multiple sentences or paragraphs and requires pattern recognition, logical reasoning, and common-sense reasoning.A mention-level GCN with path reasoning models entity interactions and infers relations; the survey reports that this approach outperforms the state of the art on DocRED.Source: §7 · p. 21
8 Conclusion
The survey reviews causality extraction methods, datasets, and evaluation metrics, then identifies challenges and potential directions. It highlights domain-specific pretraining and graph models as promising for causality extraction.
Survey scope
- The survey covers six benchmark datasets, evaluation metrics, approaches for different causality forms, and three challenges: multiple causalities, data deficiency, and document-level extraction.It identifies constructing a causal knowledge base as a fundamental task toward applications including clinical decision-making, biomedical discovery, emergency management, and news topic references.Source: §8 · p. 22
Extraction requirements
- Explicit and implicit extraction depend on linguistic keywords, patterns, and features, while intra- and inter-sentential extraction depend on input-instance preparation.The survey distinguishes these as critical steps for the respective causality forms.Source: §8 · p. 22
Future direction
- Domain-specific pretraining combined with graph models is presented as a promising direction for causality extraction.The survey notes that general-purpose pre-trained models may not perform well in specialized domains, while graph convolutional networks have advantages in complex texts.Source: §8 · p. 22