Source-linked AI summary
Classification and Clustering of Arguments with Contextualized Word Embeddings
Nils Reimers, Benjamin Schiller, Tilman Beck, Johannes Daxenberger, Christian Stab, Iryna Gurevych
TL;DR
Open-domain argument search requires identifying topic-dependent arguments and clustering semantically similar arguments, especially across unseen topics. The paper evaluates ELMo and BERT for both tasks, introduces a more realistic similarity corpus, and reports improved results across classification and clustering benchmarks.
Problem
Open-domain argument search must handle topic-dependent classification and fine-grained similarity, but clustering is understudied and available resources are limited.
Method
The paper evaluates contextualized ELMo and BERT representations for argument classification and clustering, including similarity metrics with agglomerative clustering across multiple datasets.
Results
Contextualized embeddings improve argument classification and supervised argument similarity, with fine-tuned BERT reaching a new state-of-the-art on the IBM dataset.
Takeaways & Limitations
The findings support the feasibility of open-domain argument search while showing that contextualized representations can address topic-dependent classification and similarity.
Takeaways & Limitations
The evaluated agglomerative clustering cannot model arguments that belong to multiple overlapping clusters, and end-to-end evaluation requires more realistic datasets.
Abstract
from arXiv · showhide
We experiment with two recent contextualized word embedding methods (ELMo and BERT) in the context of open-domain argument search. For the first time, we show how to leverage the power of contextualized word embeddings to classify and cluster topic-dependent arguments, achieving impressive results on both tasks and across multiple datasets. For argument classification, we improve the state-of-the-art for the UKP Sentential Argument Mining Corpus by 20.8 percentage points and for the IBM Debater - Evidence Sentences dataset by 7.4 percentage points. For the understudied task of argument clustering, we propose a pre-training step which improves by 7.8 percentage points over strong baselines on a novel dataset, and by 12.3 percentage points for the Argument Facet Similarity (AFS) Corpus.
1 Introduction
The paper frames open-domain argument search as topic-dependent argument classification followed by clustering, focusing on contextualized embeddings for both tasks. It targets challenges caused by unseen topics, semantic variation, and differently phrased arguments.
- Open-Domain Argument Search: Open-domain argument search combines topic-dependent argument classification with clustering similar arguments across sources.The clustering step aggregates repeated or semantically similar arguments for the same topic.
- Challenges: Unseen topics require systems to understand both the topic and the searched sources beyond lexical matching.Topic vocabulary can differ substantially, such as between “net neutrality” and “school uniforms”.
- Challenges: Clustering must distinguish fine-grained semantic nuances between differently phrased arguments addressing the same topic.The example arguments both concern equal Internet access while using different wording.
- Approach: ELMo and BERT generate context-sensitive representations that adapt word meanings to sentence context and are evaluated across four datasets.The study examines cross-topic argument classification and clustering using the UKP, IBM, and argument-clustering datasets described in the introduction.
- Contributions: The paper reports improved classification and clustering with contextualized embeddings, including fine-tuned models that reduce the gap to human performance.It also introduces a novel corpus for aspect-based argument similarity.
2 Related Work
Prior work addressed argument classification and clustering mainly with feature-based methods, traditional embeddings, or recurrent models. This paper applies contextualized embeddings to more realistic cross-topic settings, where training and test distributions differ substantially.
- Prior Work: Argument mining research spans discourse structure, argument quality assessment, evidence retrieval, and similarity-based clustering.The related work organizes these efforts around classification and clustering tasks relevant to open-domain argument search.
- Argument Classification: Earlier classification systems used feature-based models, GloVe embeddings, BiLSTMs, attention, or topic-aware LSTM cells.These approaches showed the importance of topic information but retained weaknesses in argument precision or recall.
- Argument Classification: Low classification performance can propagate errors into later clustering of extracted arguments.The paper therefore targets argumentative-sentence precision and recall with contextualized language models.
- Argument Clustering: Earlier clustering work primarily combined feature-based approaches with word2vec or GloVe and often used curated or topic-specific data.The paper instead evaluates argument clustering with relevant and non-relevant arguments across many topics.
- Contextualized Embeddings: Contextualized embeddings represent a target word according to its sentence context, unlike traditional embeddings that assign one vector per word.The paper examines ELMo and BERT for cross-topic argument mining.
- Cross-Topic Setting: Cross-topic evaluation is challenging because training and test distributions are vastly different, and ELMo and BERT had mainly been tested on comparable distributions.The paper investigates whether these models adapt to this additional challenge.
3 Datasets
The paper evaluates argument classification and clustering with existing corpora and introduces the UKP ASPECT Corpus for more realistic similarity judgments. The new corpus is built from search-engine arguments and includes noisy, potentially unrelated pairs.
- Dataset Scope: No end-to-end open-domain argument-search dataset is available, so classification and clustering are evaluated independently.The study uses separate resources for the involved steps.
- Argument Classification: The UKP corpus contains 25,492 sentences from 400 documents across eight controversial topics, labeled pro, con, or no argument.It supports sentence-level topic-dependent argument classification.
- Argument Classification: The IBM corpus contains evidence and no-evidence annotations for sentences associated with 118 topics from different debate portals.Each topic-sentence pair received annotations from ten crowd-workers.
- Argument Clustering: Topic-dependent argument clustering is understudied, with few resources for judging whether arguments address the same aspect.Similarity must account for aspect granularity, context dependence, and multiple aspects.
- Argument Clustering: The AFS Corpus contains 6,000 curated argument pairs from three topics, rated from 0 for different topics to 5 for complete equivalence.Its curated construction excludes noise and non-relevant arguments.
- UKP ASPECT Corpus: The UKP ASPECT Corpus uses arguments retrieved by ArgumenText for 28 technology and society topics, with pairs sampled to balance similarity diversity.Pairs were labeled as having no, some, or high similarity, with an additional unrelated option and seven assignments per pair.
- UKP ASPECT Corpus: Expert agreement on a 50-pair subset was Krippendorff’s α = 0.43 for binary distance and 0.47 for weighted distance.Agreement between experts and the crowd-worker gold standard was α = 0.54 and 0.55, respectively.
4 Argument Classification
The classification experiments replace traditional embeddings with ELMo and BERT, with fine-tuned and topic-integrating BERT models producing the strongest cross-topic results on UKP and IBM. Topic information particularly improves recall when incorporated into fine-tuned BERT, although its effect depends on corpus composition.
- UKP Corpus: Fine-tuned BERT-large improves UKP F1 by 12pp over the bilstm baseline and by around 4pp over bilstmBERT.Using an intermediary BiLSTM layer for BERT hurts performance.
- UKP Corpus: Including the topic increases UKP F1 by approximately 14.5pp for BERT-base and 13pp for BERT-large.The gains come mainly from recall, which increases by at least 21pp for positive and negative arguments.
- IBM Corpus: On IBM, BERT and ELMo embeddings improve accuracy by around 2.7pp and 3.7pp over GloVe, while fine-tuned BERT-base is 7pp higher.BERT-base differs from BERT-large by only +1pp.
- IBM Corpus: BERT-basetopic and BERT-largetopic improve IBM accuracy by 8pp over the baseline and represent a new state-of-the-art.Topic-integrating IBM ELMo and BERT models provide little improvement over their BiLSTM counterparts.
- Corpus Composition: Topic information has a smaller effect on IBM fine-tuned BERT because every sentence contains its related topic, unlike only 20% of UKP sentences.The IBM corpus therefore provides less need for additional topic information to recognize sentence relatedness.
5 Argument Clustering
The paper evaluates contextualized-embedding similarity metrics for topic-dependent argument clustering, using agglomerative clustering across the UKP ASPECT and AFS corpora. Fine-tuning improves cross-topic similarity performance, but strict clustering can reduce scores because arguments may address multiple aspects.
- Clustering Method: Argument clustering requires similarity metrics that handle aspect granularity, context-dependency, aspect multiplicity, and topic-dependent differences.The study evaluates these metrics both directly and within an agglomerative clustering pipeline.
- Clustering Method: Agglomerative hierarchical clustering uses average linkage and a train-set stopping threshold because the number of topic aspects is unknown in advance.The authors also tested k-means and DBSCAN, but agglomerative clustering generally performed better in preliminary experiments.
- Results: ELMo and BERT embeddings did not improve over averaged GloVe embeddings on the AFS corpus, while Tf-Idf achieved the best performance there.This contrasts with the ASPECT Corpus, where the relative performance of unsupervised metrics differed.
- Results: 7.8pp improvement was observed on the UKP ASPECT Corpus after fine-tuning BERT on some topics, while AFS performance improved by 11pp over the previous state-of-the-art.On ASPECT, dissimilar-argument identification was on-par with human performance, while similar-argument identification reached an F-score of .67 versus .75 for humans.
- Results: Cross-topic evaluation on AFS produced .57 Spearman correlation, still significantly above the best unsupervised method.The cross-topic setup trains on two topics and evaluates on the third.
- Results and Analysis: Clustering reduced ASPECT performance by up to 7.64pp because strict partitioning assigns each argument to exactly one cluster despite arguments addressing multiple aspects.The dataset also violates transitivity in 376 of 1,714 cases (21.9%), making strict partitioning suboptimal for ASPECT.
6 Conclusion
Open-domain argument search requires both identifying arguments for unseen topics and determining whether arguments on the same topic are similar. Contextualized embeddings substantially improve classification and supervised similarity, while clustering remains limited by overlapping aspects and current dataset design.
- Open-domain argument search combines argument identification for unseen topics with similarity-based aggregation.
- .53 F1-score improves over the previous .27 for identifying pro-arguments in cross-topic classification.Integrating topic information into BERT’s transformer network contributes an additional 13 percentage points over the setup without topic information.
- About 16% of annotated argument pairs were noisy and did not address the target topic.
- A supervised BERT similarity function significantly improves over unsupervised methods even with little training data.The low unsupervised scores indicate that fine-grained semantic nuances, rather than lexical overlap, determine argument similarity.
- Agglomerative clustering introduces errors because arguments can address multiple aspects and belong to multiple clusters.The authors identify overlapping argument clustering and more realistic end-to-end datasets as priorities for future work.
A.1 UKP ASPECT Corpus: Amazon Mechanical Turk Guidelines and Inter-annotator Agreement
The UKP ASPECT Corpus annotations were collected through a screened and paid Amazon Mechanical Turk study using worker qualifications and a test of argument-pair judgments.
- UKP ASPECT annotations were collected through Amazon Mechanical Turk crowdsourcing.
- Workers had to be US-located, have more than 100 approved HITs, and maintain an acceptance rate of at least 90%.
- Workers were paid the US federal minimum wage of $7.25 per hour.
- Qualification required passing a twelve-question test involving argument pairs.
A.2 AFS Corpus: Detailed Results
The AFS detailed-results section reports full similarity-method results and specifies correlation metrics alongside within-topic and cross-topic evaluation procedures; the annotation guidelines ask workers to judge aspect similarity for sentence pairs.
- Table 5 reports Pearson correlation r and Spearman’s rank correlation ρ for argument similarity on the AFS dataset.
- Within-topic evaluation uses 10-fold cross-validation, whereas cross-topic evaluation trains on two topics and evaluates on the third.
- Annotators judge whether each sentence pair argues about the same aspect for the specified topic.
- The annotation interface includes a “Different Topic/Can’t decide” option for off-topic or unintelligible sentences.