Causality Extraction based on Self-Attentive BiLSTM-CRF with Transferred Embeddings

1. Introduction

• Existing approaches face cross-domain limits, domain-knowledge demands, feature-engineering effort, and cascading errors from separate pair extraction and relation classification.Pattern-based approaches may have poor cross-domain applicability; feature engineering relies heavily on manual feature selection.Source: §1 · pp. 2–3
• The proposed causality tagging scheme formulates extraction as sequence labeling and models causal triplets directly instead of splitting the task into subtasks.A causal triplet consists of two event entities and their relation.Source: §1 · p. 3
• The tag2triplet algorithm handles multiple and embedded causal triplets in the same sentence, addressing the tagging scheme’s inability to represent overlapping relations.The earlier scheme assumes an entity belongs to one triplet and has a unique tag.Source: §1 · p. 3
• SCITE uses a BiLSTM-CRF with Flair embeddings transferred from a large corpus to address data insufficiency.The paper identifies difficulty training a strong deep-learning model without prior knowledge when data are insufficient.Source: §1 · pp. 4–5
• SCITE adds multihead self-attention to capture long-range dependencies between causes and effects.The paper motivates this mechanism by noting that cause and effect can be far apart; self-attention permits direct connections between arbitrary words.Source: §1 · pp. 4–5
• The method achieves significant and consistent improvement compared with other baselines.The introduction attributes this result to extensive experiments and further analysis.Source: §1 · p. 5

2.1. Causality Tagging Scheme

• The seven-tag inventory uses BIO position markers and cause, effect, embedded-causality, or other roles.“O” marks words irrelevant to causality components; the remaining tags distinguish beginnings and inside positions for causes, effects, and embedded causality.Source: §2.1 · pp. 5–6
• Embedded causality marks an event that has different causal roles in different triplets.In the example, chronic inflammation is a cause in one triplet and an effect in another.Source: §2.1 · p. 5
• The scheme represents word position and causal-event semantic roles with separate signs.The example sentence is annotated by labeling its causal event entities separately.Source: §2.1 · pp. 5–6

2.2. From Tag Sequence to Causal Triplets

• Tag2triplet converts a tag sequence into final causal triplets, with simple and complex causality treated as distinct patterns.The authors introduce the two pattern types to explain the algorithm.Source: §2.2 · p. 6

2.2.1. The Case of Simple Causality

• Simple causality has either one cause or effect without embedding, or multiple causes and effects without embedding where every triplet shares a cause or effect.The two cases are distinguished by the counts of cause, effect, and embedded-causality tags.Source: §2.2.1 · p. 7

2.2.2. The Case of Complex Causality

• Complex causality includes embedded causality, or multiple causes and effects without embedding when at least one triplet shares neither with another.The latter pattern requires at least one triplet that does not share the same cause or effect with any other triplet.Source: §2.2.2 · p. 8

2.2.3. Tag2triplet Algorithm

• The algorithm counts causal-tag degrees, classifies the sequence, forms candidate triplets, checks degree and conjunction rules, then selects a shortest-distance valid combination.Candidate triplets are generated from causal entities; coordinating conjunctions such as “and” guide whether causes form separate triplets with the same effect.Source: §2.2.3 · pp. 8, 10

2.3. SCITE

• The SCITE architecture is presented as a sequence-labeling pipeline whose input sentence S has length n.The section introduces character representations, contextual embeddings, BiLSTM, self-attention, and CRF components.Source: §2.3 · p. 10

2.3.1. CNN for Character Representations

• A character CNN applies filters to character windows and max-over-time pooling to produce each word’s character representation.The resulting character features are combined with pretrained word embeddings and transferred Flair embeddings before the BiLSTM.Source: §2.3.1 · pp. 10–12

2.3.2. Transferring Contextualized Representations Learned from Large Corpus

• To address data insufficiency, SCITE transfers Flair embeddings derived from a forward and backward character language model trained on a 1-billion-word corpus.The two output hidden states are concatenated into a word embedding, then concatenated with character representations and pretrained word embeddings for the BiLSTM input.Source: §2.3.2 · p. 12

2.3.3. BiLSTM

• A BiLSTM combines forward and backward hidden states at each position to use both past and future sentence context.The section notes that a one-directional LSTM ignores future information.Source: §2.3.3 · p. 13

2.3.4. Multihead Self-Attention

• SCITE applies multihead self-attention to BiLSTM outputs to learn dependencies between causalities, then combines attention and BiLSTM representations for tag prediction.Attention heads use separate learned projections and parallel scaled dot-product attention; their outputs are concatenated.Source: §2.3.4 · pp. 14–15

2.3.5. CRF

• SCITE’s CRF jointly decodes labels for the whole sentence, choosing the valid tag sequence with the maximum score using Viterbi decoding.The CRF models correlations between adjacent tags.Source: §2.3.5 · pp. 15–16

3.1.1. Dataset

• The corpus extends SemEval 2010 Task 8 annotations to include additional causal triplets and contains 5,236 sentences, including 1,270 with at least one triplet.The training set has 4,450 sentences and 1,570 triplets; the test set has 804 sentences and 296 triplets.Source: §3.1.1 · p. 16

3.1.2. Evaluation

• A predicted causal triplet counts as correct only when it precisely matches a labeled triplet, and experiments report mean results and standard deviations across five runs.Precision, recall, and F1-score are the evaluation metrics.Source: §3.1.2 · p. 17

3.1.4. Baselines

• The comparison includes Rules+Bayesian and CausalNet pipeline baselines, CNN and BiLSTM taggers, and BiLSTM-CRF variants using ELMo, BERT, or Flair embeddings.The Flair comparisons also include a variant with task-trained character representations from a CLSTM.Source: §3.1.4 · pp. 18–20

3.2. Experimental Results

• Table 3 compares pipeline, CNN-based, BiLSTM-based, and contextual-embedding tagging methods, with SCITE listed in the final part.The supplied passages describe the table’s organization but do not include its result values, so they do not establish which method performs best.Source: §3.2 · pp. 20–21, Table 3 · p. 21
• SCITE outperforms all other models with a test-set F1-score of 0.8455.The paper identifies this as evidence of the proposed method’s effectiveness.Source: §3.2 · p. 21
• BiLSTM-based sequence tagging models outperform CNN-based models, potentially because LSTM captures global word context and learns semantic representations of causality.The paper attributes the advantage to more efficient capture of global word context.Source: §3.2 · pp. 21–22
• Flair-BiLSTM-CRF improves 6.32% over BiLSTM-CRF, compared with ELMo and BERT increases of 6.28% and 4.55%, respectively.The authors interpret the results as indicating that contextualized character-level word embeddings are more suitable for causality extraction.Source: §3.2 · p. 22
• Under SCITE, the proposed tagging scheme yields a better result than the general scheme, 0.8455 versus 0.7682.The general scheme lacks an “Emb” tag and therefore cannot correctly identify embedded causality.Source: §3.2 · p. 22
• Although embedded causalities are relatively few in the test set, one identification error can affect the correct extraction of multiple triplets.The paper illustrates this consequence in Fig. 3.Source: §3.2 · p. 22

4.1. Error Analysis

• Most tag errors are confusions among cause, effect, and outside tags; the authors suggest insufficient annotated data may contribute.Source: §4.1 · p. 23
• SCITE ranks first in F1-score for cause, effect, and embedded-causality tags, while embedded-causality F1 is lower, with only 110 training instances.Most embedded-causality test tags are mistaken for cause or effect, lowering embedded-causality recall.Source: §4.1 · p. 23

4.2. Ablation Analysis

• Flair embeddings provide the most significant component improvement; MHSA helps especially without Flair, and character features add a slight increase.The ablation analysis reports that all components positively contribute to SCITE’s performance.Source: §4.2 · pp. 23–24
• 33.28% F1-score improvement over SCITE-Flair is reported for transferred Flair embeddings with 10% of training data.The paper reports this result in its training-data-size analysis.Source: §4.2 · p. 24, Figure 10 · p. 25

4.3. Analysis of Multihead Self-Attention

• Models with MHSA perform better than models without MHSA at arbitrary causality distances, and MHSA significantly improves performance when causality distance exceeds 10 compared with shorter-distance cases.The test set’s maximum causality distance is 24, and its distance groups have an approximate 2:2:1 ratio.Source: §4.3 · pp. 25–26

4.4. Case Study

• The simple-causality example contains five triplets, but none of the compared models extracts them all; the authors note difficulty learning phrase-level patterns.The cited example concerns recognizing “superficial or underground water” as a complete semantic unit and the phrase “as well as.”Source: §4.4 · p. 26
• The authors suggest that tagging models may need slightly more training data to learn patterns connecting two causal components.Source: §4.4 · p. 26
• In the complex embedded-causality example, only SCITE captures all dependencies and extracts all three causal triplets among the compared models.The authors state that embedded causality brings difficulty and ambiguity to learning.Source: §4.4 · p. 26

5.1. Pattern-Based Methods

• Rule-based pattern methods can transfer poorly across domains and require domain knowledge and substantial time to formulate rules.They use patterns and linguistic analysis to extract causal relations.Source: §5.1 · p. 28

5.2. Methods Based on the Combination of Patterns and Machine Learning

• Pattern-and-machine-learning systems typically extract candidate causal pairs before classifying them, creating a pipeline vulnerable to cascading errors and costly feature engineering.The reviewed approaches use clues or templates and statistical, semantic, or grammatical features to filter candidate pairs.Source: §5.2 · pp. 28–29

5.3. Methods Based on Deep Learning Techniques

• Prior deep-learning work includes causal-relation classification and models that identify linguistic expressions of causality.The review also describes work addressing common-sense reasoning about causality.Source: §5.3 · pp. 29–30
• In contrast, this method targets extraction of common-sense causal triplets rather than only classifying relations or identifying linguistic expressions.The review identifies these as distinctions from prior deep-learning approaches.Source: §5.3 · p. 30
• It handles multiple and embedded causal triplets in one sentence without splitting sentences, avoiding cascading errors as in Dasgupta et al..Source: §5.3 · p. 30

• SCITE formulates causality extraction as sequence tagging and uses a self-attentive BiLSTM-CRF with transferred Flair embeddings and multihead attention to learn cause-effect dependencies.Source: §6 · p. 30
• Experimental results demonstrate the effectiveness of the proposed method.Source: §6 · p. 30
• Insufficient high-quality annotated data still limits SCITE's performance to some extent.Source: §6 · p. 30
• Future work proposes developing annotated datasets from multiple sources and combining SCITE with distant supervision and reinforcement learning to improve performance without a high-quality annotated corpus.Source: §6 · pp. 30–31