PaperlayerView exact revision on arXiv

Source-linked AI summary

Argumentation Mining in User-Generated Web Discourse

Ivan Habernal, Iryna Gurevych

arXiv:1601.02403v5cs.CL

TL;DR

Argumentation mining needs methods that handle noisy, varied user-generated Web discourse while connecting computational analysis with argumentation theory. This paper adapts and empirically tests an argumentation model, builds a 340-document corpus of about 90,000 tokens, and evaluates sequence-labeling methods, finding the task feasible but challenging.

  • Problem

    Argumentation mining lacks broadly applicable approaches for varied, noisy Web discourse, theory-grounded models, and reliable labeled corpora.

  • Method

    The paper adapts a normative argumentation model through reliability-focused annotation studies and applies machine-learning methods to identify argument components.

  • Results

    340 documents and approximately 90,000 tokens were annotated, while the best system achieved Macro-F1 0.251 and outperformed the baseline.

  • Takeaways & Limitations

    Argumentation mining in user-generated Web discourse is feasible but challenging, and the released corpus and software support further research.

  • Takeaways & Limitations

    The model simplifies argumentation by omitting explicit relations between argument components, and Toulmin’s framework has recognized clarity and applicability criticisms.

Abstract

from arXiv · show

The goal of argumentation mining, an evolving research field in computational linguistics, is to design methods capable of analyzing people's argumentation. In this article, we go beyond the state of the art in several ways. (i) We deal with actual Web data and take up the challenges given by the variety of registers, multiple domains, and unrestricted noisy user-generated Web discourse. (ii) We bridge the gap between normative argumentation theories and argumentation phenomena encountered in actual data by adapting an argumentation model tested in an extensive annotation study. (iii) We create a new gold standard corpus (90k tokens in 340 documents) and experiment with several machine learning methods to identify argument components. We offer the data, source codes, and annotation guidelines to the community under free licenses. Our findings show that argumentation mining in user-generated Web discourse is a feasible but challenging task.

1. Introduction

Argumentation has been studied across many disciplines, with recent work linking it to human reasoning. The article introduces this broader intellectual context for computational analysis.

  • Argumentation has been studied since Aristotle across philosophy, psychology, communication studies, cognitive science, logic, linguistics, computer science, and education.
  • Recent research has claimed that argumentation drives humans to perform reasoning.
  • The supplied introduction includes the authors’ institutional affiliations and the manuscript’s submission, revision, and acceptance dates.

Original text

The article targets information overload in controversial user-generated Web content by mining argument structure across domains and registers. It develops an empirically tested argumentation model, constructs a large annotated corpus, and evaluates machine-learning methods for extracting argument components.

  • Motivation: Argumentation mining addresses information-seeking needs that opinion mining, question answering, and summarization cannot directly satisfy.The motivating use case is helping users make personal decisions about controversial topics from lengthy Web discussions.
  • Research gap: Existing Web argumentation approaches are often restricted to particular domains or registers, motivating methods that generalize across user-generated discourse.The paper identifies domain and register restriction as a central research gap.
  • Research scope: The article examines user-generated Web discourse from several domains and registers to study how argumentation is communicated in different contexts.The research questions also address model properties, complexity versus annotation reliability, Web-specific phenomena, and controversial topics.
  • Corpus: The corpus covers educational controversies and multiple Web registers, including article comments, discussion forums, blog posts, and professional newswire articles.Target domains include homeschooling, single-sex education, and mainstreaming.
  • Annotation: Two extensive annotation studies address noisy retrieval and produce 990 annotated documents before deeper argumentation analysis.The source data do not permit assumptions about topical relevance or argumentation content.
  • Corpus and analysis: 340 documents containing approximately 90,000 tokens were annotated with a micro-level argumentation model, yielding substantial agreement and supporting hand analysis of prevalent Web phenomena.The paper presents the corpus as the largest annotated argumentation-mining corpus at the time of publication.
  • Experiments: Machine-learning experiments compare feature configurations for extracting argument structure in in-domain and cross-domain settings, with data and software released under free licenses.The released resources include annotated data and experimental software.

2. Theoretical background

The theoretical background frames argumentation as persuasion involving structured components, while noting that no universally accepted theory unifies its logical, dialectical, and rhetorical dimensions. The article therefore adopts Toulmin’s original micro-level model for analyzing single-author arguments.

  • Definitions: Argumentation is presented as persuasion or an intentional effort to influence another person's mental state through communication.The discussion distinguishes this broader view from definitions focused only on giving reasons.
  • Definitions: An argument is typically defined as a claim supported by reasons, with the standpoint being central to argumentation analysis.The claim denotes what is being argued about.
  • Theoretical diversity: No unitary argumentation theory is universally accepted across the logical, dialectical, and rhetorical dimensions of argumentation.This lack of consensus motivates using models that abstract textual language into conceptual components and relations.
  • Scope: The article studies micro-level argumentation, focusing on the structure of a single argument produced by a single author rather than dialogue-level processes.Macro-level models emphasize dialogue, whereas the selected perspective treats argumentation as a product.
  • Toulmin model: Toulmin’s original model represents an argument through six components, each with a distinct role in the argument structure.The model is used later in annotation studies and experiments.
  • Toulmin model: The six components are claim, data or grounds, warrant, backing, qualifier, and rebuttal.They respectively express the assertion, evidential foundation, inference justification, warrant support, certainty degree, and defeating conditions.

3. Related work in computational linguistics

Prior computational-linguistic work covers diverse annotation models, domains, datasets, and classifiers, but often reports restricted scope, limited reliability information, or artificially controlled data. The article positions its contribution as a reliable, broad-coverage treatment of argumentation in real Web discourse.

  • Scope of related work: Related work spans argumentation mining, stance detection, persuasion, online dialogue, and computer-supported argumentation.The review first discusses annotation studies and then classification of components, schemes, and relations.
  • Annotation studies: Prior corpora include scientific publications, essays, blogs, Wikipedia, tweets, Greek texts, and legal materials, often with domain- or register-specific constraints.Examples include restricted argumentative essays, user-generated Greek texts, and topic-specific Wikipedia discussions.
  • Annotation limitations: Several studies report incomplete annotation documentation or limited reliability evidence, while AraucariaDB is criticized for missing guidelines, unreported reliability, and small size.Reported agreement varies across studies, including κ 0.69 for claims and justifications and κ 0.48 for Walton-scheme annotations.
  • Annotation studies: Annotation studies differ substantially in argumentation model, domain, corpus size, and inter-annotator agreement.Table 1 summarizes these dimensions across prior studies.
  • Classification experiments: Classification experiments use methods including Naive Bayes, maximum entropy, decision trees, kernel methods, and multi-class SVMs for identifying argumentative categories.Park and Cardie report MacroF1 0.69 for classifying three proposition types, while Stab and Gurevych report 0.72 macro-F1 for four sentence categories.
  • Positioning: The article emphasizes adapting argumentation scholarship to register-independent analysis and prioritizing reliable annotation.Its stated novelty is to connect theoretical models with empirical argumentation mining on broader Web data.

4. Annotation studies and corpus creation

The study details reproducible procedures for selecting, annotating, curating, and evaluating a new corpus for argumentation mining. It addresses research gaps including domain restrictions, weak connections to argumentation theories, and insufficient reporting of reliability or annotation schemes.

  • The corpus-development process covers data selection, annotation, curation, and evaluation to support reproducible argumentation-mining research.The authors provide a detailed overview because annotation schemes and evaluation procedures are not yet widely established.
  • The new corpus responds to domain restrictions, missing links to argumentation theories, and underreported reliability or annotation guidelines in prior work.

4.1 Topics and registers

The study focuses on controversial educational topics and samples multiple Web registers to capture diverse viewpoints and linguistic variation. Its register set combines user-generated sources with professionally edited newswire articles.

  • Topics: Educational controversies were chosen because they encompass broad subtopics and viewpoints from researchers, practitioners, parents, students, and policymakers.The authors assume this diversity creates linguistic variability that challenges NLP.
  • Topics: The selected topics include homeschooling, public versus private schools, redshirting, and prayer in schools.Redshirting means intentionally delaying an age-eligible child’s kindergarten entry to allow additional emotional and physical maturation.
  • Registers: The corpus spans comments, forum posts, blog posts, and newswire articles as four distinct registers.Newswire articles are professionally written and edited, whereas the other sources are mainly user-generated.

4.2 Raw corpus statistics

The raw corpus contains 694,110 tokens across 5,444 documents, with document length and paragraph distributions varying by register. Comments and forum posts are shorter, while articles and blogs are longer and more paragraph-rich.

  • 694,110 tokens are distributed across 5,444 documents in the raw corpus.
  • Comments and forum posts average fewer than 300 tokens, whereas articles and blogs exceed 400 tokens and average 9.2 paragraphs.

4.3 Annotation study 1: Identifying persuasive documents in forums and comments

The first annotation study filters noisy forum and comment data by distinguishing on-topic persuasive documents from non-persuasive material. Three annotators labeled 990 documents, with moderate agreement and 524 documents selected as persuasive gold data.

  • Motivation: A preliminary sample showed that many controversy-related documents contain no argumentation, including unrelated texts and topic-related but non-argumentative contributions.Such documents are unsuitable for fine-grained argumentation analysis.
  • Task design: The study defined a binary document-level task distinguishing persuasive from non-persuasive documents.The non-persuasive category includes off-topic texts, story sharing, and unrelated dialogue acts.
  • Annotation: 990 documents were annotated by three near-native English annotators, reaching 0.59 Fleiss’ κ.Final labels were assigned by majority voting.
  • Results: 524 of 990 documents, or 53%, were labeled on-topic persuasive and formed the gold data persuasive corpus.
  • Disagreement analysis: Disagreements commonly involved implicit stance, sarcasm, irony, required common-ground knowledge, and judgments of topic relevance.Interpretations depended on readers’ familiarity, cultural background, and source or discussion context.
  • Results: Moderate agreement varied across topics, with prayer in schools and private versus public schools attracting more persuasive discourse than redshirting and mainstreaming.The authors note that the persuasiveness definition may be somewhat fuzzy.

4.4 Annotation study 2: Annotating micro-structure of arguments

The study adapts Toulmin’s model for annotating micro-level argument structure in user-generated Web discourse, balancing coverage of observed phenomena with annotation reliability.

  • Model selection: The annotation study evaluates a micro-level model that captures argumentation within a single document.The study focuses on the structure of a single argument rather than relations among arguments or authors over time.
  • Model selection: Walton’s schemes were considered insufficiently comprehensive and difficult to annotate consistently in real argumentative texts.One cited study found that 37.1% of collected arguments did not fit the selected schemes, while classroom annotation revealed ambiguity and arbitrary scheme selection.
  • Model adaptation: The model omits qualifier and warrant because they are generally absent or unclear in the analyzed data.Qualifiers were not stated, while warrants were almost never explicit and their definitions were reported to fail in practice.
  • Model adaptation: Refutation was introduced to represent an author’s attack on a rebuttal, and its annotation requires an explicitly present rebuttal.This rebuttal–refutation chain corresponds to the rhetorical figure of procatalepsis.
  • Annotation scope: The scheme permits implicit claims and multiple arguments in one document while restricting annotators from creating complex argument hierarchies.Implicit claims require annotators to write the inferred authorial stance.
  • Model definition: The modified Toulmin’s model contains five non-overlapping components: claim, premise, backing, rebuttal, and refutation.Annotators select arbitrary token spans and component types together; all components are optional except the claim.

4.4.5 Annotation set-up.

The second annotation setup combines flexible token- or sentence-level labeling with unitized agreement measurement. Agreement varies substantially by register, topic, and component, and several document properties predict lower agreement.

  • Annotation setup: Annotators used sentence-level labels by default and switched to token-level annotation when sentences contained multiple components or noisy Web text made sentence boundaries unsuitable.The annotation unit was set to tokens, with pre-segmentation performed using the Stanford CoreNLP sentence splitter.
  • Annotation setup: Annotators rephrased each component into a simple statement as a sanity check for coherent discourse units.A component that could not be summarized in one statement could require splitting into multiple premises.
  • Annotation setup: The custom Web application supported token- or sentence-level granularity, logos and pathos dimensions, and component summaries.These functions allowed the same document to be annotated across different argument dimensions.
  • Agreement measurement: Krippendorff’s unitized alpha (αU) was used because it handles simultaneous disagreement over segment labels and boundaries.The measure is computed over a continuum of smallest units such as tokens, with multiple-document aggregation requiring methodological choices.
  • Agreement results: 0.48 overall αU was achieved across registers, topics, and components in the logos dimension, indicating moderate agreement under one common interpretation.The authors note that direct interpretation of agreement values lacks consensus and that comparisons with prior work are not always direct.
  • Agreement results: αU was 0.60 for comments and 0.09 for articles, while claim and premise agreement was higher in comments and forum posts than in articles and blogs.Claim and premise agreement was 0.59 and 0.69 in comments and forum posts, versus 0.22 and 0.24 in articles and blogs.
  • Agreement results: Topic-level αU ranged from 0.68 for prayer in schools and 0.44 for private versus public schools to 0.14 for redshirting and 0.08 for mainstreaming.The spread indicates that some topics were substantially more difficult to annotate than others.
  • Disagreement analysis: Longer comments, longer blog paragraphs, and lower readability in the private-versus-public-schools domain correlated with lower agreement.No significant correlations were observed for sentence coverage or average sentence length, and the authors avoid drawing a general conclusion.

4.4.9 Analysis of annotated corpus from argumentation research perspective.

The annotated corpus reveals implicit, topic-sensitive, and often weakly balanced argumentation in user-generated Web discourse. The analysis also identifies limits of the annotation model for rhetorical and non-standpoint-driven texts.

  • Corpus-level patterns: 48% of claims are implicit, while explicit claims are rephrased in only 3% of documents.Readers must often infer the standpoint from reasons rather than repeated explicit claims.
  • Stance and opposition: 85% of documents omit opposing sides; rebuttals occur in 8%, followed by refutations in 4%.Multiple rebuttals and refutations occur in 3% of documents.
  • Hedging and rhetorical style: Hedging varies by topic: about 30–35% of homeschooling and mainstreaming claims hedge, compared with 15% for prayer in schools and about 10% for private versus public schools.Prayer-in-schools claims are often direct, imperative, and sometimes offensive.
  • Model suitability: The modified Toulmin model is suitable for short persuasive texts but is not expressive enough for rhetoric-heavy discourse in articles and blog posts.Stylistic devices, narratives, direct and indirect speech, and interviews often lack a distinguishable logos function.
  • Model suitability: Texts without a clear standpoint are difficult to annotate because fitting balanced reasons into the framework can cause disagreements.The authors suggest that paragraph-level annotation might address this problem.

4.5 Summary of annotation studies

The paper reports two annotation studies examining user-generated Web argumentation at document and micro levels, with moderate agreement for document-level persuasiveness. The second study selects and extends a model for detailed argument annotation.

  • Study overview: Two annotation studies addressed argumentation in user-generated Web content at different levels of detail.The studies covered document-level persuasiveness and micro-level argument structure.
  • Document-level study: Document-level annotation covered 990 comments and forum posts and achieved moderate inter-annotator agreement, with Fleiss’ π 0.59.The paper argues for distinguishing persuasiveness at the document level.
  • Micro-level study: The micro-level study selected an argumentation model, extended it using pre-study observations, and established an annotation setup.Its focus was detailed annotation of argument components.

5. Experiments

The experiments evaluate sentence-based argument-component identification across random, in-domain, and cross-domain settings. Performance is feasible but limited by strict token-level evaluation, rare classes, boundary errors, and domain-sensitive preprocessing and features.

  • Evaluation scenarios: The experiments compare random, in-domain, and cross-domain validation scenarios for argument-component identification.Cross-domain validation trains on five domains and tests on the sixth.
  • Overall results: Human annotators achieved Macro-F1 0.602 and accuracy 0.754, with lower performance on rebuttal and refutation than on claims, premises, backing, and non-argumentative text.Excluding rebuttal and refutation, human Macro-F1 would be 0.707.
  • Overall results: Macro-F1 reached 0.251 with rich feature sets 01234 and 234, significantly outperforming the baseline and other feature sets.Non-argumentative text reached about 0.7 F1 even with baseline features.
  • Cross-domain results: Embeddings generalized relatively well across domains, making performance comparable to in-domain results while most other feature combinations performed poorly.The authors leave investigation of stronger vector representations for future work.
  • Error analysis: About half of the instances in each class were misclassified as non-argumentative, while Refutation-B and Refutation-I were misclassified as Premise-I in 19% and 27% of cases.The authors relate the refutation confusion to its context-dependent role and similarity to premises.
  • Discussion and limitations: The strict 11-class token-level metric penalizes boundary mismatches like wrong component types, while preprocessing tools trained on newswire data may lose accuracy on user-generated content.The best system achieved 0.30 with Krippendorf’s αU and 0.32 on boundary similarity.

6. Conclusions

The study finds that argumentation theories can be adapted to user-generated Web discourse, but model performance and annotation reliability vary with corpus and discourse properties. It introduces a corpus, annotation resources, and machine-learning methods for identifying argument components.

  • Conclusions: The modified Toulmin model balances expressiveness and annotation reliability, although its effectiveness depends on register, topic, document length, and literary structures.These properties influenced annotator agreement and the model’s ability to capture argumentation.
  • Conclusions: Online argumentation includes rhetorical questions, figurative language, narratives, and fallacies that lack established theoretical counterparts.Attempts to model some of these phenomena in the pathos dimension did not achieve satisfactory agreement.
  • Conclusions: Because no single argumentation theory fits all Web data, model and annotation-scheme selection should reflect task requirements and corpus properties.The authors recommend a Toulmin-based scheme for short documents such as comments and forum posts.
  • Conclusions: A sequence-labeling system significantly outperformed the baseline, achieving 0.251 overall macro-F1 versus 0.156.Word-embedding features performed best in cross-domain evaluation, reaching 0.209 macro-F1.
  • Conclusions: The corpus contains 340 documents and approximately 90k tokens annotated with a modified Toulmin model, following an initial study of 990 documents.The authors also provide annotation guidelines, annotated data, source code, and system outputs for error analysis.

Appendix A: Raw corpus compilation

The raw corpus was compiled semi-automatically from six controversial educational topics and four user-generated Web registers, combining automated extraction with manual selection and cleaning.

  • Appendix A: Raw corpus compilation: Websites containing relevant article comments and forum discussions were manually identified across six topics, without preferring a particular platform or data source.Texts were then extracted automatically with minimal pre-selection and cleaning.
  • Appendix A: Raw corpus compilation: Article and blog texts were manually selected and checked for argumentative relevance, formatting, and boiler-content removal.Paragraph formatting was retained because paragraphs play an important role in argumentative discourse.
  • Appendix A: Raw corpus compilation: The raw corpus comprised material from 117 unique source domains, with document distributions reported by topic and register.The ten most frequent domains are listed separately in Table 1.

Appendix B: Examples of annotated documents from the second annotation study

The annotated examples show how the scheme represents claims, premises, rebuttals, backing, and emotional appeals across noisy Web comments. They also illustrate token-level annotation and separate logos and pathos dimensions.

  • Annotated examples: The homeschooling example contains a claim, rebuttal, premises, and emotional appeal represented in both logos and pathos dimensions.The annotation captures figurative phrases such as “child abuse” and “ruin your child’s whole life” as emotional appeals.
  • Annotated examples: A second homeschooling comment uses backing and premises to support an implicit argument against homeschooling, including claims about social skills and inadequate instruction.The example draws on the writer’s stated experience as an educator, student, parent, and admissions officer.
  • Annotated examples: The examples preserve strongly noisy user-generated text, including incorrect capitalization and punctuation that required token-level annotation.Automatic sentence splitting could not process the example properly.
  • Annotated examples: A public-versus-private-schools comment pairs a premise criticizing public-school spending with the claim “TAKE YOUR KIDS PRIVATE IF YOU CARE AS I DID.”The premise alleges that additional teacher funding does not improve children’s education.
  • Annotated examples: Another public-school example is annotated entirely in the pathos dimension because it appeals to emotions through the phrase “send children to a pig farm.”Its implicit claim opposes public schools and uses rhetorical language rather than an explicitly structured logos argument.
Loading 1601.02403v5…