Source-linked AI summary
A Large-scale Dataset for Argument Quality Ranking: Construction and Analysis
Shai Gretz, Roni Friedman, Edo Cohen-Karlik, Assaf Toledo, Dan Lahav, Ranit Aharonov, Noam Slonim
TL;DR
Argument quality ranking lacks large, point-wise annotated resources and robust ways to convert binary crowd judgments into continuous scores. The paper constructs IBM-Rank-30k, evaluates scoring functions and quality dimensions, and develops a BERT-based ranker that outperforms several baselines and performs competitively on prior data.
Problem
Argument quality assessment needs larger point-wise datasets and principled ways to derive continuous quality scores from binary crowd annotations.
Method
The paper creates IBM-Rank-30k, compares MACE-P and weighted-average scoring, analyzes quality dimensions, and evaluates BERT-based ranking methods.
Results
BERT-V improves over Bi-LSTM GloVe by .4-.6 Pearson and .2-.6 Spearman points, while fine-tuning BERT adds .2-.4 points for both measures.
Takeaways & Limitations
IBM-Rank-30k’s scale, collection process, scoring analysis, and BERT-based method provide resources for further argument-quality-ranking advances.
Takeaways & Limitations
The authors could not fully address Durmus, Ladhak, and Cardie because that work was published after submission.
Abstract
from arXiv · showhide
Identifying the quality of free-text arguments has become an important task in the rapidly expanding field of computational argumentation. In this work, we explore the challenging task of argument quality ranking. To this end, we created a corpus of 30,497 arguments carefully annotated for point-wise quality, released as part of this work. To the best of our knowledge, this is the largest dataset annotated for point-wise argument quality, larger by a factor of five than previously released datasets. Moreover, we address the core issue of inducing a labeled score from crowd annotations by performing a comprehensive evaluation of different approaches to this problem. In addition, we analyze the quality dimensions that characterize this dataset. Finally, we present a neural method for argument quality ranking, which outperforms several baselines on our own dataset, as well as previous methods published for another dataset.
1 Introduction
Argument quality assessment is a prominent computational-argumentation task with applications in decision making, argument search, and writing support. This work introduces a large point-wise quality dataset, compares ways to derive scores from binary annotations, and proposes BERT-based ranking methods.
- Argument quality assessment supports automated decision making, argument search, and writing support.
- 30,000+ arguments form a carefully annotated point-wise quality dataset, IBM-Rank-30k, five times larger than prior released point-wise data.The dataset contains around 30k arguments and uses active collection, strict length limits, and extensive quality control.
- The work compares approaches for inducing continuous quality labels from binary crowd annotations.Binary judgments are used instead of direct continuous ratings to reduce variation in annotator-specific scales.
- Argument quality spans dimensions including clarity, relevance, impact, grammar, and other linguistic phenomena.
- The proposed BERT-based neural method outperforms several baselines on the authors’ data and is comparable to or better than prior methods on released data.
2 Related Work
Earlier research assessed argument quality through point-wise scores, pairwise comparisons, or context-specific impact measures. The paper situates its dataset and task among these alternatives while distinguishing overall quality from narrower notions such as impact.
- Argument quality assessment has longstanding theoretical and practical roots, with subjectivity identified as a core challenge.
- SwanRank contains 5.3k arguments labeled with real-valued interpretability scores in [0, 1].
- Pairwise approaches compare arguments with similar stance, and related datasets include UKPRank with 1k arguments and UKPConvArgAll with 16k arguments and argument-pairs.
- Durmus et al. provide over 47k claims across 471 topics, modeling impact in argumentative context rather than overall quality independently.
- Toledo et al. introduced IBMRank with 5.3k point-wise arguments and IBMPairs with 9.1k labeled argument-pairs for civic-engagement use cases.
- The authors note that Durmus, Ladhak, and Cardie’s work appeared after their submission and therefore was not fully addressed.
3 IBM-Rank-30k Dataset
IBM-Rank-30k was built through controlled crowd collection and binary quality annotation across 71 controversial topics. The resulting corpus contains 30,497 arguments gathered under length, contributor, and annotation-quality controls.
- The dataset construction process covers argument collection, quality annotation, and derivation of continuous labels from binary annotations.
- Arguments were collected for 71 controversial topics through Figure Eight, with 8.6% supplied by expert annotators.
- Contributors wrote one supporting and one contesting argument per topic using original language, with extra payment tied to subsequent quality judgments.
- 30,497 arguments were collected from 280 contributors, with each contributor limited to six arguments per topic and each argument restricted to 35–210 characters.
- Each argument received binary quality judgments from 10 annotators, who also marked the argument’s stance toward the topic.
- Annotation quality was monitored with hidden stance test questions, annotator-reliability scores, and a selected pool of 600 experienced crowd annotators.Annotators failing more than 20% of hidden test questions were removed; typically 5%–10% were removed per sub-task.
- The average valid annotator-reliability score for quality annotations was 0.12, while stance task-average-κ reached 0.83.
4 Deriving an Argument Quality Score from Binary Annotations
The paper converts binary crowd judgments into continuous argument-quality scores using MACE-P and WA, then compares their distributions, agreement with pair-wise judgments, and consistency across annotator splits.
- Scoring functions: Binary annotations are converted into continuous scores because annotators may apply subjective scales when directly assigning continuous quality values.Each scoring function estimates the likelihood of a positive quality label between 0 and 1.
- Scoring functions: MACE-P uses an item-response model with annotator reliability, whereas WA computes a reliability-weighted average to reduce unreliable judgments.WA is designed to provide an intuitive, gradual form of data cleansing.
- Score distributions: WA produces a gradual distribution concentrated toward higher scores, while MACE-P produces a U-shaped distribution concentrated near both extremes.WA reaches values near 0 or 1 mainly under strong annotation consensus, whereas MACE-P assigns probabilities to both labels.
- Agreement with pair-wise annotation: 55% of disagreement pairs favored MACE-P over WA according to pair-wise annotators, while simple-average was omitted after performing worse in subsequent comparisons.Among differing pairs, annotators selected the MACE-P-preferred argument in 61% of comparisons with simple-average and the WA-preferred argument in 59%.
- Agreement with pair-wise annotation: As score differences increased, precision against pair-wise annotation increased for both functions, with WA reaching a perfect match above a 0.75 difference.For WA, fewer pairs were filtered for low annotator agreement in the three bins above 0.25, suggesting stronger agreement when WA strongly favors one argument.
- Split annotations consistency: Splitting each argument’s ten annotations into two five-annotation sets yielded good score correlation for both functions, although reduced information can harm reliability estimation.WA achieved Pearson and Spearman correlations of 0.42 and 0.36, respectively, while MACE-P achieved 0.42 for both.
- Conclusion: The authors prefer WA for deriving non-binary scores but retain both functions for neural-method evaluation because experiments did not identify a clear overall winner.The choice of scoring function affects score distributions and the performance of models trained on those scores.
5 Analysis of Quality Dimensions
The paper analyzes argument quality through a multidimensional model, showing that higher-quality bins score higher across dimensions and differ most strongly in Global Relevance and Effectiveness.
- Quality-dimensions model: The analysis uses a quality-dimensions model that decomposes holistic argument quality into fine-grained properties.The model derives 15 sub-dimensions from a broad literature survey, while the experiment excludes five dimensions and evaluates ten.
- Annotation procedure: The experiment samples 100 arguments across five equally populated WA-quality bins and has three experts rate ten dimensions on a 1–3 scale.Annotators were unaware of the original quality bins, and dimension scores were averaged for each argument.
- Results: Across all dimensions, arguments in the highest-quality bin score above middle-bin arguments, which score above lower-bin arguments.The bin ordering persists despite the complexity of the annotation task.
- Results: The largest differences between quality bins 5 and 1 occur for Global Relevance at 0.72 and Effectiveness at 0.64.These dimensions therefore show the strongest separation between the highest- and lowest-quality groups in this analysis.
6 Argument Quality Ranking
The paper evaluates several baselines and BERT-based models for argument quality ranking on IBM-Rank-30k and related datasets. Fine-tuning and topic information improve correlations on IBM-Rank-30k, while cross-dataset results vary by dataset and labeling setup.
- BERT-V improves over Bi-LSTM GloVe by .4-.6 Pearson points and .2-.6 Spearman points on IBM-Rank-30k.
- Fine-tuning BERT improves over BERT-V by .2-.4 points for both Pearson and Spearman correlation.
- Adding the topic produces a statistically significant .1-.2-point improvement for both correlation measures and quality-score methods.The reported significance is p ≪0.01.
- Performance rises toward the quality extremes, reaching .71-.73 Pearson and .67 Spearman when evaluating the bottom and top 10% of the test set.The cut-off analysis uses the BERT-FTTOPIC model and considers symmetric top-and-bottom percentiles.
- On SwanRank, BERT-FTTOPIC improves the average RRSE by .8 points over the selected prior cross-domain result, improving on 3/4 topics.Performance decreases on the evolution topic.
- On UKPRank, the methods obtain Pearson correlations comparable to several prior methods but worse Spearman correlations.The comparison is less direct because UKPRank point-wise scores were induced from pairwise labels, whereas the proposed method does not require labeled pairs.
7 Learning to Represent Quality
The paper examines how BERT representations change after fine-tuning for argument quality ranking. An example using the token “people” compares pre-trained and fine-tuned embeddings across high- and low-quality arguments.
- Fine-tuning updates BERT’s weights, changing the contextual representations of tokens for the new task.
- The analysis compares pre-trained and BERT-FT embeddings of “people” using arguments sampled from five WA quality bins.The sample contains 20 arguments, with 10 from the lower and 10 from the higher quality ranges.
- Figure 3 visualizes these embeddings with 2D t-SNE projections before and after fine-tuning, distinguishing high- and low-quality arguments.The left panel uses BERT’s pre-trained model and the right panel uses the fine-tuned model.
8 Conclusions and Future Work
The paper concludes that IBM-Rank-30k advances argument quality ranking through a large point-wise dataset, annotation-score analysis, and a BERT-based ranking method. It also identifies Global Relevance and Effectiveness as the most indicative quality dimensions and proposes further investigation.
- IBM-Rank-30k contains 30,497 point-wise quality-labeled arguments, making it the largest such dataset reported and five times larger than previous releases.
- The work addresses continuous-label induction from binary annotations through an extensive comparison of two common approaches.
- The BERT-based method outperforms several baselines on the authors’ dataset and achieves promising results on other datasets.
- Global Relevance and Effectiveness are the most indicative dimensions of overall quality scores in the IBM-Rank-30k analysis.Future work will examine how these dimensions affect overall quality and whether models capture them effectively.