Source-linked AI summary
Knowledge-Augmented Language Models for Cause-Effect Relation Classification
Pedram Hosseini, David A. Broniatowski, Mona Diab
TL;DR
The paper investigates commonsense knowledge augmentation for causal reasoning and cause-effect classification, where such augmentation has task- and domain-dependent effects and its impact on classification has been little studied. It verbalizes ATOMIC20 and GLUCOSE for continual pretraining of BERT and RoBERTa, reporting improved cause-effect classification and better results than baselines on TCR. These gains are reported without new model architecture or quality-enhanced fine-tuning data.
Problem
The impact of commonsense knowledge augmentation varies by domain and downstream task, and its effect on cause-effect relation classification has received limited study.
Source: §1 · p. 1Method
The authors automatically verbalize ATOMIC20 and GLUCOSE, then continually pretrain BERT and RoBERTa using masked language modeling on the resulting text.
Source: §2 · p. 2Results
The continually pretrained models significantly boost cause-effect pair classification and the best GLUCOSE-pretrained model significantly outperforms the baseline and Ning et al.’s joint inference framework on TCR.
Source: §1 · p. 2, §4 · p. 4Takeaways & Limitations
Cause-effect classification performance can improve using automatically verbalized commonsense knowledge without new model architecture or quality-enhanced fine-tuning data.
Source: §1 · p. 2The experiments do not use margin-based loss, extra regularization loss, or quality-enhanced fine-tuning data, leaving room for further model improvement.
Source: §4 · p. 5
Abstract
from arXiv · showhide
Previous studies have shown the efficacy of knowledge augmentation methods in pretrained language models. However, these methods behave differently across domains and downstream tasks. In this work, we investigate the augmentation of pretrained language models with commonsense knowledge in the cause-effect relation classification and commonsense causal reasoning tasks. After automatically verbalizing ATOMIC2020, a wide coverage commonsense reasoning knowledge graph, and GLUCOSE, a dataset of implicit commonsense causal knowledge, we continually pretrain BERT and RoBERTa with the verbalized data. Then we evaluate the resulting models on cause-effect pair classification and answering commonsense causal reasoning questions. Our results show that continually pretrained language models augmented with commonsense knowledge outperform our baselines on two commonsense causal reasoning benchmarks, COPA and BCOPA-CE, and the Temporal and Causal Reasoning (TCR) dataset, without additional improvement in model architecture or using quality-enhanced data for fine-tuning.
1 Introduction
The paper examines how commonsense knowledge augmentation affects causal reasoning and cause-effect classification, where its impact varies by domain and task and causal classification has received limited study. It uses automatically verbalized knowledge for continual pretraining and reports improved cause-effect pair classification without new model architecture or quality-enhanced fine-tuning data.
1 Introduction
- The effect of commonsense knowledge injection on cause-effect relation classification has received limited study, despite its task- and domain-dependent performance.Source: §1 · p. 1
- The work studies pretrained language models augmented with commonsense knowledge on the less-investigated task of cause-effect relation classification.Source: §1 · p. 1
- A simple masked-language-modeling framework with automatically verbalized commonsense knowledge significantly boosts cause-effect pair classification without new architecture or quality-enhanced fine-tuning data.Source: §1 · p. 2
- The authors publicly release their knowledge-graph verbalization code and continually pretrained models.Source: §1 · p. 2
2 Method
The method verbalizes ATOMIC20 and GLUCOSE as text, checks the text for obvious language errors, and uses it for continual masked-language-model pretraining of BERT and RoBERTa. The resulting models are evaluated on downstream benchmarks.
2 Method
- The framework verbalizes commonsense knowledge graphs, continually pretrains BERT and RoBERTa with masked language modeling, and evaluates the resulting models on benchmarks.Source: §2 · p. 2
2.1 ATOMIC20 20 to Text
- ATOMIC20 triples are converted to text by joining subjects, relation-specific human-readable templates, and targets.The data are filtered to remove duplicates, none-valued targets, and triples containing blanks, yielding 782,848 triples.Source: §2.1 · p. 2
2.2 GLUCOSE to Text
- GLUCOSE causal relations are verbalized by joining their spans with templates for causal connectives, then split randomly into 90% training and 10% development data.GLUCOSE contains ten causal-explanation dimensions, including specific statements and general rules.Source: §2.2 · pp. 2–3
2.3 Checking Grammar
- LanguageTool checks verbalized ATOMIC20 and GLUCOSE text for obvious grammatical mistakes and spelling errors.The authors leave comparisons with other data-to-text and grammar-checking methods for future research.Source: §2.3 · p. 3
2.4 Continual Pretraining
- BERT-large-cased and RoBERTa-large are continually pretrained with masked language modeling on ATOMIC20 and GLUCOSE separately for up to 10 epochs, using early stopping on evaluation loss.Training uses a maximum sequence length of 304 and batch size 128.Source: §2.4 · p. 3
3 Experiments
The experiments evaluate commonsense causal reasoning and cause-effect relation classification on TCR, COPA, and BCOPA-CE. They compare continually pretrained models with pretrained baselines using task-specific input formats and classifiers.
3.1 Benchmarks
- TCR, COPA, and BCOPA-CE are selected to test causal reasoning and cause-effect classification, with BCOPA-CE described as a more challenging COPA-inspired benchmark with unbiased token distributions.Source: §3.1 · p. 3
- COPA development data are used for fine-tuning and test evaluation on COPA-test and BCOPA-CE, while TCR models are fine-tuned and evaluated on its train and test splits.Results are averaged across eight random-seed runs; TCR uses 10 epochs, batch size 8, and learning rate 2e-5.Source: §3.1 · p. 4
3.2 Models and Baseline
- Pretrained BERT-large-cased and RoBERTa-large serve as baselines; COPA tasks use a multiple-choice head, and TCR uses R-BERT for relation classification.COPA and BCOPA-CE instances are converted to SWAG format; TCR instances use special tokens to mark event boundaries.Source: §3.2 · p. 4
4 Results and Discussion
Continually pretrained models outperform baselines on TCR and COPA, with gains also reported on COPA’s harder questions. For BCOPA-CE, adding question-type prompts improved performance over the baseline and prior models fine-tuned on the same data.
4 Results and Discussion
- The GLUCOSE-continually-pretrained model significantly outperforms the TCR baseline and Ning et al. (2018)’s joint inference framework.Table 1 reports TCR accuracy results, but the supplied passages do not include the table’s values.Source: §4 · p. 4, Table 1 · p. 4
- Continually pretrained models significantly outperform COPA-test baselines; GLUCOSE models use about 70k training points and perform on par with or slightly better than ATOMIC20 models trained on more data.ATOMIC20 event-only models use about 121k points, while models trained on all three relation types use about 780k; event-only models achieve almost the same performance as the all-types models.Source: §4 · p. 4, Table 2 · p. 5
- On COPA, BERT-Large and RoBERTa-Large achieved performance on par with BERT-SocialIQA without crowdsourced or manually corrected verbalizations.The authors also report room for improvement through model-improvement methods or fine-tuning with quality-enhanced BCOPA data.Source: §4 · p. 5
- The models significantly outperform baselines on both Easy and Hard COPA questions, including the Hard questions that do not contain superficial cues.The supplied passages describe the split results but do not provide their accuracy values.Source: §4 · p. 5, Table 3 · p. 5
4.1 BCOPA-CE: Prompt vs. No Prompt
- Adding question-type prompts improved BCOPA-CE performance, and the best prompted model outperformed Han and Wang (2021)’s b-l-aug and b-l-reg models fine-tuned on the same data.The prompts were “It is because” for cause questions and “As a result” for effect questions.Source: §4.1 · p. 5, Table 4 · p. 5
5 Conclusion
The paper introduces a framework that augments pretrained language models with commonsense knowledge from automatically verbalized ATOMIC20 and GLUCOSE. The augmented models outperform the original models on cause-effect pair classification and commonsense causal reasoning questions.
5 Conclusion
- The framework augments pretrained language models with commonsense knowledge from automatically verbalized ATOMIC20 and GLUCOSE.Source: §5 · p. 5
- Commonsense knowledge-augmented models outperform the original pretrained language models on cause-effect pair classification and commonsense causal reasoning questions.Source: §5 · p. 6
- The authors identify previously proposed model-improvement methods and unbiased fine-tuning datasets as possible next steps for enhancing the knowledge-augmented models.Source: §5 · p. 6
A Contribution of Augmented Knowledge
The analysis examined whether semantic overlap between augmented knowledge and benchmark examples might help explain correct predictions unique to the augmented models. Similar GLUCOSE and COPA examples were found, but the authors caution that this overlap does not establish the cause of performance gains.
A Contribution of Augmented Knowledge
- The analysis compared augmented-model-only correct predictions with ATOMIC2020 and GLUCOSE using sentence-embedding cosine similarity, retaining pairs at 50% similarity or higher.The examples were drawn from samples correctly predicted across all random-seed runs by the best model but missed by the baseline.Source: A Contribution of Augmented Knowledge · p. 8
- 3,588 GLUCOSE pairs had at least 0.5 cosine similarity with a COPA sample, and 253 pairs had at least 0.6.GLUCOSE contains 70,730 training samples; Table 5 shows correctly classified COPA samples and their most semantically similar GLUCOSE entries.Source: A Contribution of Augmented Knowledge · p. 8, Table 5 · p. 8
- Similar event contexts, such as seeing an unfamiliar car and feeling curious, may relate to correct COPA predictions, but the authors do not claim this similarity caused the gains.They suggest explicit causal statements may help models express causal knowledge, whether or not it was already encoded.Source: A Contribution of Augmented Knowledge · p. 8