Source-linked AI summary
Shrome at Touch\'e: Soft-Vote Ensembling and Counter-Causal Augmentation for Causality Extraction
Roham Zendehdel Nobari, Shayan Sooratgar
TL;DR
Causality extraction must distinguish asserted causal claims from news sentences that mention a cause-effect pair while denying the relation. The paper combines separate models for detection, span extraction, and polarity, using soft-vote ensembling and counter-causal data augmentation. On held-out evaluation, it reports strong extraction and polarity results, while noting that development-set choices were made without held-out internal validation.
Problem
Counter-causal statements are largely neglected in causality extraction, although distinguishing asserted causes from denied ones is essential to representing causal knowledge faithfully.
Source: §1 · p. 1Method
The system uses separate detection, extraction, and polarity models, combining span-gated detection, emission-level CRF ensembling, and nine-pattern LLM augmentation for counter-causal training data.
Source: §1 · p. 2, §7.2 · p. 9, §11 · p. 13Results
Granularity-adjusted F1 0.728 was the highest extraction score in the organizers’ final evaluation, while held-out test F1 was 0.869 on detection and macro F1 was 0.817 on polarity.
Source: §11 · p. 13Takeaways & Limitations
The results demonstrate a system that handles counter-causal claims across detection, span extraction, and polarity on CCNC, with the highest extraction score in the organizers’ final evaluation.
Source: §11 · p. 13Nearly all discrete design choices were selected on the same development split without held-out internal validation, and the largest dev-to-test gap was 12.0 pp for extraction.
Source: §10 · p. 12
Abstract
from arXiv · showhide
Touch\'e 2026 extends causality extraction to counter-causal claims: news sentences whose surface form appears causal but whose meaning denies the causation, as in "It is falsely believed that X caused Y." A system that relies on surface cues such as "caused" or "led to" will accept such a sentence as causal and give it the wrong polarity. On the Countercausal News Corpus (CCNC), the task has three subtasks: deciding whether a sentence is causal (detection), locating its cause and effect spans (extraction), and labeling its polarity as procausal, counter-causal, or uncausal. We build one model per subtask. Detection is a fine-tuned classifier with a single cross-task rule that uses the extracted spans to remove false positives. For extraction, we ensemble three RoBERTa-large BILOU+CRF taggers by averaging their token-level scores before decoding, rather than voting on the spans each tagger produces. For polarity, where labeled counter-causal examples are scarcest, we add training sentences generated by a large language model prompted with nine patterns of counter-causal expression adapted from Hagen et al., keeping only those that pass automatic structural checks. On the held-out CCNC test set, the system reaches F1 0.869 on detection and macro-F1 0.817 on polarity, and in the organizers' final causal-only evaluation of extraction it scores granularity-adjusted F1 0.728, the highest extraction score among all submissions including the organizers' baseline. The development split is used only for component selection and the ablations reported in the paper.
1. Introduction
Counter-causal news can mention an apparent cause-effect pair while denying that the causation occurred, a distinction needed to represent causal knowledge faithfully. The paper addresses this gap with separate detection, extraction, and polarity models, alongside prior work on causal-news corpora and extraction and augmentation methods.
1. Introduction
- Counter-causal sentences contain an apparent cause-effect pair while denying the relation, so surface lexical cues can misclassify both causality and polarity.The task separates binary detection, cause/effect span extraction, and three-way polarity labeling.Source: §1 · p. 1
- The system combines three subtask models, emission-level soft-vote extraction ensembling, nine-pattern LLM augmentation for polarity, and span-based gating of detection positives.The paper reports held-out test F1 0.869 for detection and macro F1 0.817 for polarity, plus the highest extraction score in the organizers’ final evaluation: granularity-adjusted F1 0.728.Source: §1 · p. 2
2. Related Work
- CCNC extends prior causal-news resources by including counter-causal statements, a category largely neglected in causality-extraction research [1].The earlier Causal News Corpus shared tasks [6] focused on causal extraction without this category.Source: §1 · p. 1, §2 · p. 2
- Encoder–CRF taggers using BILOU labels are an established span-extraction approach; BoschAI’s CASE 2023 system ranked first on Subtask 2.That system used pipe-concatenated labels to represent up to three cause–effect–signal relations per sentence.Source: §2 · pp. 2–3
- Prior low-resource labeling work includes EDA, label-aware token operations, LLM-generated examples, NLI verification [17], and task-adaptive pre-training.The paper notes that sentence-level EDA edits can disturb token-level span boundaries.Source: §2 · p. 3
3. Dataset
The dataset section describes CCNC training data and the distinct units used for detection/extraction versus polarity. It also highlights the counter-causal class’s training imbalance and its explicit negation cues.
3. Dataset
- Detection and extraction use sentence-level data, while polarity has more rows because each candidate cause/effect pair is annotated separately.Training uses organizer-provided splits; extraction and polarity additionally use in-domain augmentation.Source: Table 1 · p. 3, §3 · p. 3
- The counter-causal class is the smallest polarity training class but shows explicit negation cues most often.The cue-prevalence comparison is measured on development data.Source: Table 2 · p. 3, §3 · p. 3
- 2,433 augmentation rows raise the counter-causal class from 886 to 3,319 training rows.The added examples come from nine-pattern LLM augmentation.Source: Table 2 · p. 3
4. System Overview
Three independently trained models handle detection, extraction, and polarity, with extracted spans gating detection through a single rule. The dev results favor the stacked detector over the listed baselines.
4. System Overview
- The stacker flips detector positives to uncausal when extraction finds no spans, while standalone detection, extraction, and polarity models otherwise run independently.The rule flips 46 dev sentences: 31 existing false positives are corrected and 15 new false negatives are introduced.Source: §5.1 · pp. 4–5, §4 · p. 4, Figure 1 · p. 4
5.1. Architecture, training, and the cross-task stacker
- 0.8946 dev causal-class F1 is the conjunctive span-threshold-one rule’s maximum, a 2.6 pp gain over the standalone detector.The rule was selected by grid search over ten candidates on the dev split.Source: §5.1 · pp. 4–5
- The dev gain has limits: the rule reduces causal recall, was selected among ten candidates on 340 sentences, and its test-set transfer cannot be quantified.Causal recall on the 207 gold causal sentences fell from 0.9130 standalone to 0.8406 stacked.Source: §5.1 · p. 5
5.2. Baselines and dev results
- The span-gated stacker is the production configuration and adds the final increment over fine-tuned BERT-base, which clears every detection baseline by a wide margin.Table 3 reports detection dev binary F1; its last row is the stacked production system.Source: §5.2 · p. 5, Table 3 · p. 5
6. Extraction
The extraction system uses three-layer BILOU+CRF taggers, combining three seeds by averaging CRF scores before decoding. On development data, emission-level voting improves exact-match F1 over a single seed, while the NLI verifier’s final change is within noise.
6.1. Per-model architecture: three-layer BILOU+CRF
- Each extraction seed uses a shared RoBERTa-large encoder with three independent BILOU heads and CRFs; external data, EDA, and overlap-F1 early stopping jointly lift single-seed dev exact F1 to 0.5742.CCNC extraction uses one entity label rather than separate cause and effect labels; no training sentence requires more than three layers.Source: §6.1 · pp. 5–6
6.2. Emission-level soft-vote across three seeds
- Averaging three seeds’ CRF emissions and transition matrices before per-layer Viterbi decoding gains 1.07 pp exact F1 over span-level voting in a matched panel.The production panel combines two base-recipe seeds with one TAPT seed; TAPT helped only one of four tested seeds and is retained for emission diversity, not as a method-level endorsement.Source: §6.2 · pp. 6–7
6.3. NLI span verifier
- The post-hoc NLI verifier keeps spans above an entailment threshold; at τ=0.1 it removes three truncated gold-span fragments, trading a marginal exact-F1 gain for a comparable overlap-F1 loss on dev.Thresholds τ≥0.2 degrade F1 monotonically by removing borderline true spans.Source: §6.3 · pp. 7–8
6.4. Final results
- 1.91 pp exact-match F1 lifts the single-seed baseline to 0.5933 in the production panel; the NLI verifier’s final step sits below per-span granularity on dev.Table 6’s Hagen et al. 0.440 is BIO-tag macro F1 on a different split, so it is only an indicative reference.Source: Table 6 · p. 8, §6.4 · p. 8
7. Polarity Classification
The polarity system uses typed cause/effect markers, nine-pattern LLM augmentation, and a four-seed BERT ensemble. Its dev macro-F1 is 0.9029 and test macro-F1 is 0.817; uncausal errors remain the largest residual class.
7.1. Architecture
- Typed PURE markers label cause and effect spans, and their addition improved dev macro F1 by 0.8 percentage points.Four independently seeded BERT-base models average their softmax outputs for the final three-way label.Source: §7.1 · p. 8, Figure 5 · p. 9
7.2. The nine-pattern counter-causal augmentation
- LLM-generated examples conditioned on nine counter-causal patterns expand the counter-causal training class from 886 to 3,319 rows.Structural checks preserve tagged content and require exactly one cause and one effect pair; their 93.2% acceptance rate is not a semantic-quality measure.Source: §7.2 · p. 9
7.3. Four-seed ensemble (with offline calibration)
- Per-model temperature scaling improved calibration measures but left ensemble dev macro F1 unchanged, so deployment uses T = 1.0.Temperatures were fitted on the development set as an offline analysis.Source: §7.3 · p. 10
7.4. Results
- 0.9029 dev macro F1 and 0.817 test macro F1: the four-seed BERT-base ensemble outperforms the best single seed on dev.The five-seed variant scored lower; the production ensemble row reports macro and countercausal F1.Source: §7.4 · p. 10, Table 7 · p. 10
- 21 of 502 dev sentences were uncausal examples predicted as procausal or countercausal, the largest residual error class.Procausal-countercausal confusion was five to six mistakes per direction, and the no-augmentation ablation widened that gap.Source: §7.4 · p. 11, Table 8 · p. 11
8. Held-Out Test-Set Results
The held-out CCNC evaluation used the organizers’ official TIRA evaluator, with a separate final causal-only extraction re-evaluation. The system led extraction and placed highest among teams on polarity, while detection ranked third on F1.
8. Held-Out Test-Set Results
- The dev-to-test gap was largest for extraction at 12.0 percentage points, consistent with development-set overfitting; domain shift remains unsubstantiated.The test evaluation reports extraction token IoU, which cannot separate boundary misalignment from full span misses.Source: §8 · pp. 11–12
- 0.728 granularity-adjusted F1 on final causal-only extraction was highest among submissions, and 0.817 polarity macro F1 was highest among teams.Extraction scores were 0.665 for the baseline and 0.587 for the other team; polarity was 0.778 for the other team and 0.839 for the baseline.Source: §8 · p. 12
- Detection ranked third on F1 at 0.869, with the highest recall (0.890) and lowest precision among the three systems.Source: §8 · p. 12
9. Discussion and Future Work
The discussion treats the small NLI-verifier gain as uncertified and proposes future work on polarity supervision and extraction modeling.
9. Discussion and Future Work
- The NLI verifier moved dev exact F1 from 0.5933 to 0.5953, with a 0.0019 overlap-F1 cost; the gain is within noise and was not statistically tested.The three spans removed at the selected threshold were truncated gold spans, improving exact F1 while slightly reducing overlap F1.Source: §9 · p. 12
- A rule-based negation-scope feature is proposed for polarity, motivated by negation cues in 67.8% of counter-causal versus 15.8% of procausal examples.Source: §9 · p. 12
- For extraction, proposed directions include alternative encoders, boundary-focused or span-scoring architectures, and parser-constrained span candidates.The parser-based constraint would make many one-sided clause-attachment errors structurally impossible when the parser is correct.Source: §9 · p. 12
10. Limitations
The authors identify development-set selection, limited statistical support, and single-corpus evaluation as principal validity boundaries. Generated augmentation also has unresolved pretraining-overlap and semantic-labeling risks.
10. Limitations
- Nearly every discrete design choice was selected on the same development split without held-out internal validation, and extraction had the largest dev-to-test gap at 12.0 percentage points.Source: §10 · p. 12
- Statistical support for smaller gains is thin: extraction soft-vote and detection stacker gains are single-run point estimates without variance bands.The NLI verifier’s dev exact-F1 gain is described as within noise rather than certified.Source: §10 · p. 12
- CCNC, drawn from Indian-English news outlets, is the only evaluated corpus, leaving transfer to other domains, English varieties, and genres untested.Source: §10 · p. 12
- Structural validators verify tag preservation, not semantic counter-causality, and generated outputs were not checked for overlap with dev or test splits.The authors also note that the language model may have encountered CCNC or Hagen pattern exemplars during pretraining.Source: §10 · p. 13
11. Conclusion
The system addresses the Touché 2026 causality-extraction task on CCNC with separate models and a limited cross-task interaction. It leads all submissions in the organizers’ causal-only extraction evaluation, while the authors caution that the stacker’s net transfer to test cannot be isolated.
11. Conclusion
- 0.869 binary F1 on detection, 0.817 macro F1 on polarity, and 0.728 granularity-adjusted F1 on causal-only extraction, the highest extraction score among submissions.These are held-out test-set results; the extraction comparison includes the organizers’ baseline.Source: §11 · p. 13
- The system combines three-seed emission-level soft-vote extraction, span-gated detection, and nine-pattern counter-causal augmentation adding 2,433 polarity training examples.The authors describe the contribution as empirical rather than architectural.Source: §11 · p. 13
- The authors cannot attribute net test transfer to the detection stacker without an unstacked test baseline, despite a 2.6 pp dev gain in causal-class F1.They retain the rule because precision-recall asymmetry favors precision and it did not degrade performance on tested splits.Source: §11 · p. 13
Declaration on Generative AI
The authors disclose GPT-5.4 use to generate polarity-task training examples and Claude assistance with code and writing. They state that humans reviewed the work and retain responsibility for its content.
Declaration on Generative AI
- GPT-5.4 generated the nine-pattern counter-causal training examples for the polarity task under the reported structural validation.The authors also used Claude for code-generation assistance on training and evaluation scripts and for writing assistance.Source: Declaration on Generative AI · p. 13
- Claude-assisted training and evaluation scripts were human-reviewed and tested.Source: Declaration on Generative AI · p. 13
- The authors state that all scientific decisions, analyses, and conclusions are their own and that they take full responsibility for the publication.Source: Declaration on Generative AI · p. 13
A. Training Hyperparameters
The appendix reports the extraction seed’s TAPT preparation, model configuration table scope, compute and release details, and counter-causal data-generation checks.
A. Training Hyperparameters
- Extraction seed 42 uses an encoder prepared with 50 epochs of masked-language modeling on 4,327 in-domain sentences, at learning rate 10^-4 with 15% token masking.Table 10 lists fine-tuning configurations for the three deployed subtask models; unlisted settings follow library defaults.Source: A. Training Hyperparameters · p. 16, Table 10 · p. 16
B. Reproducibility Details
- The seven deployed seeds finish fine-tuning in under twelve GPU-hours on one RTX 5090, and development-split inference takes under sixty seconds.The released repository includes version pins, checkpoints, inference and soft-vote code, prompts, generated data, and calibration.Source: B. Reproducibility Details · p. 16
- CCNC is used under the organizers’ data-use agreement for CLEF 2026 participation, while external corpora and the NLI checkpoint are used under upstream licenses and not redistributed.Source: B. Reproducibility Details · p. 16
- The 2,433 augmented examples come from training-split seeds and exemplars, with no development or test sentence used in prompts.Structural checks accepted 2,433 of 2,610 attempts (93.2%), verifying byte-equal tag preservation and exactly one cause/effect pair; the authors characterize this as a syntactic, not semantic, pass rate.Source: B. Reproducibility Details · p. 16