Source-linked AI summary
Cross-topic Argument Mining from Heterogeneous Sources Using Attention-based Neural Networks
Christian Stab, Tristan Miller, Iryna Gurevych
TL;DR
Argument mining systems often target specific text types, limiting argument search over heterogeneous Web documents. The paper proposes a crowd-applicable sentence annotation scheme and evaluates attention-based models across eight controversial topics. The attention-based network generalizes best to unseen topics and improves over vanilla BiLSTM models by 6% in accuracy and 11% in F-score.
Problem
Most argument-mining approaches are designed for specific text types and do not work well on heterogeneous sources, limiting robust argument search.
Method
The paper proposes a sentential annotation scheme for topic-oriented arguments and uses it to build data for cross-topic attention-based neural-network experiments.
Results
6% accuracy and 11% F-score relative improvements over vanilla BiLSTM models were achieved in cross-topic experiments, while inner-att+cos generalized best to unknown topics.
Takeaways & Limitations
The scheme and corpus support cross-topic argument-search experiments on heterogeneous text types, and the attention-based model better generalizes to unknown topics than vanilla BiLSTM models.
Takeaways & Limitations
The study is restricted to keyword-expressible topics and arguments consisting of individual sentences.
Abstract
from arXiv · showhide
Argument mining is a core technology for automating argument search in large document collections. Despite its usefulness for this task, most current approaches to argument mining are designed for use only with specific text types and fall short when applied to heterogeneous texts. In this paper, we propose a new sentential annotation scheme that is reliably applicable by crowd workers to arbitrary Web texts. We source annotations for over 25,000 instances covering eight controversial topics. The results of cross-topic experiments show that our attention-based neural network generalizes best to unseen topics and outperforms vanilla BiLSTM models by 6% in accuracy and 11% in F-score.
1 Introduction
Argument search addresses information needs for controversial topics, where answers require assessing stance, source, and support rather than retrieving a single factual answer. The paper introduces a heterogeneous-text annotation scheme and evaluates attention-based models across topics.
- Motivation: Argument mining supports searches for controversial or opinion-based topics that lack a single canonical answer.Users may need to assess each answer’s stance, source, and supportability.
- Research gap: Most existing argument-mining approaches target particular text types and perform poorly on different data sets and Web sources.
- Contribution: The proposed annotation scheme is designed for information-seeking argument search, heterogeneous sources, and manual application by untrained annotators.
- Corpus: The resulting corpus contains over 25,000 instances across eight controversial topics, enabling cross-topic experiments with heterogeneous text types.
- Results: 6% accuracy and 11% F-score relative improvements over vanilla BiLSTM models were obtained in cross-topic experiments.
2 Related Work
Prior argument-mining research largely models argumentative discourse within documents or retrieves structured claims from restricted sources. This paper instead targets topic-relevant arguments in arbitrary texts and cross-domain settings.
- Existing approaches: Discourse-level approaches segment argumentative units, classify their functions, or recognize relations within a single document.
- Argument search: Generic argument search frameworks rely on structured arguments from debate portals and cannot yet retrieve arguments from arbitrary texts.
- Task distinction: Stance detection identifies support or opposition regardless of whether sentences contain supporting evidence, unlike argument search.
- Cross-domain work: Earlier cross-domain experiments concern discourse-level tasks, whose models depend strongly on their original text types.
3 Annotation Scheme and Corpus Creation
The paper defines a flat, sentence-level annotation scheme for topic-oriented evidence or reasoning and constructs a crowdsourced corpus from heterogeneous Web documents. Expert comparisons and MACE filtering support the reliability and scale of the resulting annotations.
- Annotation scheme: An argument is a text span expressing evidence or reasoning that supports or opposes a given topic.
- Annotation scheme: The flat model classifies arguments independently of surrounding context, reducing the need to read large amounts of text or model relations among arguments.
- Scope: The study restricts topics to keyword-expressible controversies and arguments to individual sentences.
- Data collection: The corpus was built from eight controversial topics and Web documents retrieved through Google and filtered using the Wayback Machine.
- Data processing: 27,520 sentences remained after boilerplate removal, sentence segmentation, tokenization, part-of-speech tagging, and basic filtering.
- Quality analysis: 0.721 Cohen’s κ measured agreement between the two experts, exceeding the 0.7 threshold used for reliable results.
- Corpus creation: 25,492 gold-standard annotations remained after applying MACE with a threshold of 0.9.
4 Approaches for Identifying Arguments
The paper formulates argument identification as topic-dependent binary sentence classification and evaluates BiLSTM-based approaches that incorporate topic information through similarity features and attention.
- Argument identification classifies sentence s as an argument or no argument for topic t based on whether it gives a relevant supporting or opposing reason.
- 4.1 Bidirectional Long Short-Term Memory Network (bilstm): The baseline bilstm is a bidirectional long short-term memory network whose final representation combines forward and backward sequence contexts.Dropout and a two-unit softmax layer are used for classification.
- 4.2 BiLSTM Model with Topic Similarity Features (bilstm+cos): The bilstm+cos model adds each word’s cosine similarity to the averaged topic embedding, addressing the baseline’s failure to model sentence–topic relations.The topic representation is the average of the embeddings of topic words.
- 4.3 Inner-attention BiLSTM (inner-att): The inner-attention model learns topic-dependent importance weights over input word embeddings before applying a BiLSTM.The attention mechanism filters uninformative words according to the given topic.
- 4.4 Inner-attention BiLSTM with Topic Similarity Features (inner-att+cos): The inner-att+cos model combines inner attention with cosine similarity between each word embedding and the averaged topic embedding.This representation emphasizes topic-similar words and helps discover off-topic sentences.
5 Evaluation
The evaluation compares in-topic and cross-topic generalization, showing that attention-based models perform strongly and that combining attention with topic similarity is especially effective for unknown topics.
- Evaluation setup: The evaluation uses in-topic and cross-topic experiments to test model robustness and generalization to unknown topics.In-topic training uses 80% of each topic, while cross-topic training uses seven topics and evaluates on the remaining unknown topic.
- In-topic results: All neural approaches outperform the lr-uni baseline in in-topic experiments.The vanilla bilstm model achieves 0.727 accuracy and 0.721 F1, while inner-att reaches 0.744 accuracy and 0.741 F1.
- Cross-topic results: Cross-topic neural-model F-scores drop by 0.108 on average, while precision improves by 0.064 compared with in-topic experiments.The models recognize fewer arguments and achieve lower recall on unknown topics, but their higher precision suggests learned properties shared across topics.
- Cross-topic results: 0.693 accuracy and 0.658 F-score make inner-att+cos the best model for generalizing to unknown topics.It outperforms vanilla bilstm on all topics, achieves 0.067 higher precision, and has the lowest recall drop relative to in-topic experiments.
- Model analysis: The inner-att+cos model combines attention and topic similarity, outperforming bilstm+cos and inner-att in cross-topic settings.The attention mechanism weights word embeddings according to the topic, while cosine similarity supplies an additional topic-related feature.
- Model analysis: Attention weights emphasize topic-relevant and argumentative words, but emphasize stop words and less topic-dependent terms when the topic is irrelevant.Across topics, highly weighted words include evaluative adjectives and verbs such as “infringe,” “oppose,” and “undermine.”
- Target-topic data: Adding target-topic data increases recall but lowers precision, with inner-att+cos reaching 0.802 recall using only 30% of target-topic data.Vanilla bilstm and bilstm+cos do not reach their in-topic recall with all available target-topic data.
6 Conclusion
The paper presents an argument-search approach for heterogeneous texts, combining a general annotation scheme, a large crowdsourced corpus, and cross-topic evaluation. Its attention-based model generalizes better to unknown topics than vanilla BiLSTM models.
- Contributions: The paper introduces an annotation scheme designed for information-seeking argument search and heterogeneous texts.The scheme is intended to be reliably applied by untrained annotators to arbitrary Web texts.
- Contributions: The resulting corpus contains over 25,000 instances across eight topics and supports cross-topic experiments with heterogeneous text types.The annotations and source code for downloading sentences are made available for future work.
- Results: The attention-based model generalizes better to unknown topics than vanilla BiLSTM models.The paper reports this conclusion from both in-topic and cross-topic experiments.