Source-linked AI summary
Are You Convinced? Choosing the More Convincing Evidence with a Siamese Network
Martin Gleize, Eyal Shnarch, Leshem Choshen, Lena Dankin, Guy Moshkowich, Ranit Aharonov, Noam Slonim
TL;DR
Identifying more convincing evidence on debatable topics remains a challenging task. The paper introduces IBM-EviConv, a data set of evidence pairs, and a Siamese network for convincingness. The network outperforms prior methods on both the earlier data set and IBM-EviConv.
Problem
The paper targets the challenging task of identifying which evidence is more convincing for a debatable topic.
Source: §1 · p. 1Method
The authors construct IBM-EviConv from labeled evidence pairs and propose a Siamese network for argument convincingness.
Source: §1 · pp. 1–2Results
The Siamese network outperforms methods by Habernal and Gurevych (2016a) and Simpson and Gurevych (2018) on both data sets.
Source: §1 · p. 2Takeaways & Limitations
IBM-EviConv emphasizes homogeneous evidence pairs, which the authors believe gives learning algorithms a better chance to reveal convincingness signals beyond more trivial ones.
Source: §1 · p. 2The authors could not reproduce earlier improvements from pretraining in these experiments, and detailed architecture improvements were outside this work’s focus.
Source: §7 · p. 9
Abstract
from arXiv · showhide
With the advancement in argument detection, we suggest to pay more attention to the challenging task of identifying the more convincing arguments. Machines capable of responding and interacting with humans in helpful ways have become ubiquitous. We now expect them to discuss with us the more delicate questions in our world, and they should do so armed with effective arguments. But what makes an argument more persuasive? What will convince you? In this paper, we present a new data set, IBM-EviConv, of pairs of evidence labeled for convincingness, designed to be more challenging than existing alternatives. We also propose a Siamese neural network architecture shown to outperform several baselines on both a prior convincingness data set and our own. Finally, we provide insights into our experimental results and the various kinds of argumentative value our method is capable of detecting.
1 Introduction
This work frames argument convincingness as choosing the more convincing evidence on debatable topics. It introduces IBM-EviConv and a Siamese network, reporting that the network outperforms prior methods on both evaluated data sets.
1 Introduction
- The paper targets evidence convincingness: given evidence texts for a debatable topic, identify the more convincing ones.The introduction situates this empirical work alongside earlier analyses of factors that make arguments convincing and calls it the first work on evidence convincingness.Source: §1 · p. 1
- IBM-EviConv uses evidence-only pairs controlled for length and drawn from Wikipedia, making pairs more homogeneous in argument type, length, and writing level.More than 8,000 pairs were sent for convincingness labeling, using evidence sentences from a data set with more than 2,000 sentences across 118 topics.Source: §1 · pp. 1–2
- Unlike UPKConvArg’s same-stance pairs, IBM-EviConv includes both same-stance and cross-stance pairs, supporting comparisons within a side and between debate sides.The cross-stance task is choosing which side of a debate was more convincing.Source: §1 · p. 2
- The Siamese network outperforms methods by Habernal and Gurevych (2016a) and Simpson and Gurevych (2018) on both data sets.The paper presents this as surpassing state-of-the-art performance.Source: §1 · p. 2
2 Background
Prior work studied convincingness in full texts and individual arguments, while pairwise and pointwise learning-to-rank approaches offer different ways to compare or score arguments. The proposed Siamese network is presented as combining practical advantages with strong results without task-specific linguistic features.
2 Background
- Earlier work examined convincingness in full texts, while recent studies and UPKConvArg address convincingness of individual arguments.Earlier studies considered prior beliefs and text structure; UPKConvArg contains roughly 1,000 web-mined arguments across 16 topics, with pairwise crowd preferences.Source: §2 · p. 2
- Pointwise ranking assigns each item a score, whereas pairwise ranking predicts preferences between item pairs and can rank a list by comparing pairs.Pairwise ranking assumes transitivity or requires an approximation when preferences are not transitive.Source: §2 · p. 3
- Habernal and Gurevych’s pairwise methods provide pairwise inference only, while Simpson and Gurevych’s GPPL produces an individual convincingness score and supports scalable inference.The related work notes that graphical models can be advantageous at small training sizes such as UPKConvArg.Source: §2 · p. 3
- The proposed Siamese network achieves state-of-the-art results for convincingness classification and ranking without task-specific features or costly linguistic-feature preprocessing.The authors also describe it as efficient at inference and updatable with advances in neural-network research.Source: §2 · p. 3
3 Siamese Network
The paper adapts a Siamese network for convincingness scoring, training its shared legs on pairs and using a single leg to score an argument at inference. Its network outputs probabilities rather than RankNet’s output, with each leg implemented as a BiLSTM with attention.
3 Siamese Network
- The Siamese network has two identical parameter-sharing legs connected to a softmax, and its output is a probability rather than RankNet’s output.The authors report comparable held-out performance to RankNet in initial experiments.Source: §3 · p. 3
- During training, softmax over the pair’s convincingness outputs is compared with the pair label using cross-entropy loss.This objective maximizes the probability of identifying the more convincing argument and pushes the two outputs apart.Source: §3 · p. 3
- At inference, one leg scores a single argument by softmaxing its convincingness output against an untrained dummy output.A higher resulting probability means the network considers the argument more convincing; each leg uses a BiLSTM with attention.Source: §3 · pp. 3–4
4 The IBM-EviConv data set
IBM-EviConv is a balanced dataset of evidence pairs designed to assess convincingness across same- and cross-stance comparisons. Its annotation and quality checks yield decisive labels with measured human agreement and high triplet transitivity.
4 The IBM-EviConv data set
- The dataset contains 1,884 unique evidence sentences from almost 70 debatable topics.Source: §4 · p. 4
- After cleaning, IBM-EviConv contains 5,697 evidence pairs, split into 4,319 training and 1,378 test pairs with no topic shared across splits.Pairs are sampled within topics, and each pair contains evidence referring to the same topic.Source: §4 · p. 4
- Pairs use evidence from Wikipedia, with roughly matched lengths and either matching or opposite stances; the dataset includes 3,075 same-stance and 2,622 cross-stance pairs.The length difference within a pair is limited to 30% of the shorter evidence.Source: §4 · p. 4
- Each pair was annotated by 10 crowd labelers, and a preference counted as convincing only when at least 60% chose that evidence; less decisive pairs were removed.Labelers were asked which evidence they would rather use in a conversation where they could give only one.Source: §4 · p. 4
- The most frequent label covers only 53% of pairs, which the authors take as evidence that the dataset is balanced and lacks strong sentence-length bias.Source: §4 · p. 4
4.1 Data set quality
- Quality checks found 84% crowd–expert agreement on decisive pilot pairs, average pairwise Cohen’s Kappa of 0.33, and expected transitivity in 99% of fully annotated triplets.The expert labelers’ average pairwise Kappa was 0.38; the pilot excluded 21 indecisive pairs, leaving 84 for comparison.Source: §4.1 · p. 5
5 Experiments
EviConvNet performs similarly to the best prior systems on UKPConvArg tasks and significantly outperforms all systems on IBM-EviConv. The experiments also show that gains over baselines are larger on IBM-EviConv.
5.1 Experiments over UKPConvArgStrict and UKPConvArgRank
- On UKPConvArgStrict, EviConvNet performs similarly to the best previously known systems.Source: Table 1 · p. 5, §5.1 · p. 5
- On UKPConvArgRank, EviConvNet provides at least equal performance to GPPL opt and a statistically significant increase in Pearson’s r.The reported significance is p ≪0.01 using a one-sample two-tailed t-test.Source: Table 2 · p. 6, §5.1 · p. 6
5.2 Experiments over IBM-EviConv
- On IBM-EviConv, the sentence-length baseline has nearly the same accuracy as always choosing the most frequent label.Source: §5.2 · p. 6
- On IBM-EviConv, EviConvNet significantly outperforms all systems and improves over GPPL by 9%, compared with 1% on UKPConvArg.GPPL improves over the sentence-length baseline by 26% on IBM-EviConv, compared with 5% on UKPConvArg.Source: Table 3 · p. 6, §5.2 · p. 6
6 Analysis
The analysis examines how EviConvNet performs across preference reasons and stance and length conditions, and what distinguishes evidence labeled convincing. It finds that the model’s strengths vary by preference reason, while it generalizes to length-imbalanced pairs.
6.1 Performance across preference reasons
- EviConvNet better captured presentation, relevance, and content quality than the length baseline, but had 57% greater error on balanced-objective arguments, only 3% of the data.The length baseline was also better on reasons involving the sheer amount of supporting information.Source: Figure 1 · p. 7, §6.1 · p. 7
6.2 What makes a convincing evidence?
- Experts cited source reliability and evidence type, completeness, specificity, significance, rhetorical strength, relevance, and supporting detail; difficult pairs often left both items unconvincing or a factor unresolved.Source: §6.2 · p. 7
- Word clouds compare topic- and stopword-filtered unigram distributions: convincing evidence featured argumentation, studies, and authorities or courts; non-convincing evidence featured opinions and less-emphasized actions.Source: §6.2 · pp. 7–8, Figure 2 · p. 8
6.3 Cross vs. same stance evidence pairs
- Cross-stance pairs were not harder to classify than same- or mixed-stance pairs, and training on cross-stance pairs did not improve cross-stance test performance.Source: §6.3 · p. 8, Table 5 · p. 8
6.4 The effect of length difference
- EviConvNet achieved 0.69 accuracy on the test set, below 0.73 on balanced data, but still exceeded all other baselines, suggesting generalization across evidence pairs of different lengths.Source: §6.4 · p. 8
7 Discussion and future work
The discussion presents IBM-EviConv as a focused, challenging dataset for evaluating argument convincingness and proposes a Siamese network that outperforms state-of-the-art methods. It also identifies pretraining and architecture improvements as avenues for future work.
7 Discussion and future work
- IBM-EviConv offers a focused, more challenging dataset for evaluating methods that identify the more convincing argument.Source: §7 · p. 9
- The paper proposes a Siamese network for argument convincingness that outperforms state-of-the-art methods.Source: §7 · p. 9
- Pretraining one network leg on argument-detection data showed improvements in earlier, smaller-data experiments, but those gains were not reproduced here.The authors suggest pretraining may help more when training data are limited.Source: §7 · p. 9
- Learning-to-rank improvements and more careful architecture design were not the focus of this work and are proposed for future testing.Examples include contextualized embeddings, batch normalization, and deeper networks.Source: §7 · p. 9